Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 388.16072ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.187514ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 367.125471ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:44:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 367.147398ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 402.924403ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 402.956506ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 447.37501ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.394459ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 413.629557ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:42:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.657305ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 348.087843ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:41:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 348.11354ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 504.519622ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:40:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 504.546127ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.025275552s trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:40:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02529767s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 16.612507874s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 14:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 16.612537652s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 13.133292279s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 14:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 13.133329172s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 15.03443099s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 14:00:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 15.035031307s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.657103132s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:59:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.657132664s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.819799253s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.819825717s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 18.878361831s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 18.878397395s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.693978531s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:56:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 9.694001418s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.839542567s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:56:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.839565075s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.465316719s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:55:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.465341289s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.099047744s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:54:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.099088039s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 12.327340441s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 12.327365462s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.751325609s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.7513535s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.259239663s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.259270813s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.77319005s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:50:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.773220589s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.895398339s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:49:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.895496258s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.610495209s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.610526538s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.357802625s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:48:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.3578336s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.177173847s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:47:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 7.177223609s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.598430392s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:47:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 7.598449245s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.941832056s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.941865046s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.398119584s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.398150626s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.862729444s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.86275889s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.873143655s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.873171836s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.735825484s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:44:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 7.735850212s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 10.259170169s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:44:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 10.259203188s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.614524946s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.614556619s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.307564714s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.307593394s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 8.660226471s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:42:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 8.66025278s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.982100925s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:42:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.982130235s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.96124547s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:42:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.961283189s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.688433478s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:41:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.688456261s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.642379277s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:41:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.642397035s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.677191909s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:39:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.677219362s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.412104774s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:39:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 5.412152721s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.982689282s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:38:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.982729964s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 17.002422321s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:37:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 17.002447567s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.751949862s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:37:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.75204759s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.197489323s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:36:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.197517876s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.69086246s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:36:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.690893723s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.648285616s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:35:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 4.648313402s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 3.930510836s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:35:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.930548806s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 3.891010114s trace: https://jaeger.dev.storj.io/trace/12dbee1725a89af6 FAIL (2025-12-13 13:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=197386]: Uploading piece... Dec 13 13:20:22.157523997 (189h52m22.985232392s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157563526 (189h52m22.985271946s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.157782226 (189h52m22.985490616s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158220136 (189h52m22.98592853s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158561593 (189h52m22.986269971s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.158833133 (189h52m22.986541524s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159110305 (189h52m22.986818699s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159379461 (189h52m22.987087838s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.159627778 (189h52m22.987336156s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Uploading piece... Dec 13 13:20:22.588165740 (189h52m23.41587413s) [upload=197386]: Next returned nil segment Dec 13 13:20:22.588188840 (189h52m23.415897231s) [upload=197386]: Waiting for error group managing segments... Dec 13 13:20:22.787355936 (189h52m23.615064332s) [noise=true , address=185.167.204.205:4005 , piece_id=B5NHOBIEY4KGWHIUBAP2N6FWVIMFBSWUS4DDWQVFMKSSSB654UPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.793561088 (189h52m23.621269481s) [noise=true , address=185.167.204.205:4013 , piece_id=QNJLAFTMBQPQF7IBMT6EKMBEL5KX7VHU3L4L3ALPXYNUV2MZ43GA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.801170609 (189h52m23.628879002s) [noise=true , address=185.167.204.206:4068 , piece_id=P2FTDV3XW34Z6M73CWIURQCCQ6H5NXDNJWKBVN3GRQTDV65QLP5A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.812199494 (189h52m23.63990789s) [noise=true , address=185.167.204.206:4048 , piece_id=MOJVQUBRM36ENNJVV7EYAXQEZQIXN7S5UBP3SBHERO7LVFDZ4HIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813259194 (189h52m23.640967588s) [noise=true , address=185.167.204.205:4006 , piece_id=DFHXNKDAWUBPQ4KGMDTUQ76MJIDUAX5U7OQIPNXJYRELBIVXYVVQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.813831001 (189h52m23.641539395s) [noise=true , address=185.167.204.206:4056 , piece_id=TYQW5XG5HOLULH5R7UGMEDCTMLXOWEA3A6WEBN6EVI5UNUZ5HH2A, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814310957 (189h52m23.642019347s) [noise=true , address=185.167.204.205:4001 , piece_id=WXEZMLB4ZM2FFOB4JMCIF2X6FTB72I3EWWRTDLIZ6SXCMUTS5HGQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.814961478 (189h52m23.64266987s) [noise=true , address=185.167.204.206:4046 , piece_id=KLHWCCQLK3CAOIMW57WAFZHNMECYLGA4VRZE3LXRUJRQCKAGGBKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.815469120 (189h52m23.643177512s) [noise=true , address=185.167.204.206:4054 , piece_id=ZPZSVEBAXMRI7732XKIHUGMU2WGGCNKE34U2FZ3NHJLKCMYBIUXQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.817780190 (189h52m23.645488584s) [noise=true , address=185.167.204.206:4045 , piece_id=VQ37X765RRBAVPI4HIAPJET7VRCI2FE7FRHB45II6ETFYDYMKSTQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818186706 (189h52m23.645895101s) [noise=true , address=185.167.204.206:4038 , piece_id=MKQFEVYRSI4HGUMDZZGQEIMROWZAQDFWQYJ3TPCZE56ROFBY6GBQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.818345893 (189h52m23.646054284s) [noise=true , address=185.167.204.205:4014 , piece_id=4U6VXOGIXKAV7NJBQBD2V6ZVWAUIT557OP7AUG332UWVS5LZFSPA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:22.844792090 (189h52m23.672516847s) [noise=true , address=185.167.204.206:4051 , piece_id=A4VZAGPIVLUVOVOIGGIK7CEDRPUSV34XMUL5TFL5KFJNICAWSJ4Q, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.354540799 (189h52m25.182249185s) [noise=true , address=81.171.14.70:4033 , piece_id=5NKXLV6JVGJJJYMT7IUC3GR5LRRXGS5F6Y6IEMX24DIQT7FTXXIA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.369904036 (189h52m25.197612417s) [noise=true , address=81.171.14.70:4039 , piece_id=YV7FP35KY6TXYDKBEOV6RCHCSJWR4TQCFEB43GK3AZP4C776BCZQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.378758619 (189h52m25.206467012s) [noise=true , address=81.171.14.70:4047 , piece_id=EQFLTLGTZBKQ47MWFLSBCEHDKAWMZSGJHZPBAN67EANSP2LR7HYQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.384761037 (189h52m25.212469431s) [noise=true , address=89.149.201.147:4006 , piece_id=LE3FPTMI6GWHTCCWL5H4MZPEGLHCTBWHWCACVUOIQWCU62JYIRCQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.495107931 (189h52m25.322816323s) [noise=true , address=81.171.14.70:4029 , piece_id=EOO6MF4FIN6OLOXXBS7GDFF4IMK2YYH75CFI4OGNNOYRCAHCSAKQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:24.833114410 (189h52m25.660822801s) [noise=true , address=89.149.201.147:4009 , piece_id=SAES2PXHWVMC22L3DKLBYHAI3ZHKYKGYOULL4ZZ2AB2U53B5URVA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.076006180 (189h52m25.903714572s) [noise=true , address=89.149.201.147:4008 , piece_id=HL2EBKXU5RWRNZP7D3MUOTRJK2YJTADFQS6C4IF7CGEUF3PLOIMQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.109256616 (189h52m25.936964999s) [noise=true , address=89.149.201.147:4022 , piece_id=3VH5FONJYYDQQ4AK74JQTD26TTLRZPTRCZH7WLHXX5Q7Y2OXXIZA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.121296573 (189h52m25.949004965s) [noise=true , address=81.171.14.70:4034 , piece_id=XZ72ODCL37AYLPMMFTCT4NJROVRCLTEM2XN6SQVG3G5NTZRJDWDA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.144861937 (189h52m25.972570318s) [noise=true , address=89.149.201.147:4012 , piece_id=5LDACT5ZGR54I4LNLZQHNXUY2VZQ5XIWI5AYUM5CZHCOOQIBGPRQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.859441967 (189h52m26.687150358s) [noise=true , address=81.171.14.70:4026 , piece_id=R7B2VE64HYLI7A2XVWDFTIIWAVLT6I6E75ZCAMTDH77R7BW2Q7IQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:25.932179545 (189h52m26.759887944s) [noise=true , address=89.149.201.147:4001 , piece_id=GDNVHYCJJDXLUDF77L6BP4IMIMR6KRZ2FTZJ6BG5TMAZGIW75PHQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228904886 (189h52m27.056613283s) [noise=true , address=89.149.201.147:4024 , piece_id=7MCMBHNW4L7JDB4ATRHAM3GWVFAYH34LWFNX5O7C6EXIB34BNWRA, seg_pos={0 0}, upload=197386]: Done uploading piece. err: <nil> Dec 13 13:20:26.228954088 (189h52m27.056662491s) [seg_pos={0 0}, upload=197386]: Segment reached optimal threshold of 26 pieces. Dec 13 13:20:26.229843002 (189h52m27.057551404s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: read tcp 159.223.143.39:53228->81.171.14.70:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.229857857 (189h52m27.05756626s) [noise=true , address=81.171.14.70:4027 , piece_id=ANUN37MZAFWJ3FTQDU6KXRNZOLN2D62GA6AL65ENUEXX2XJEE3GQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230208353 (189h52m27.057916754s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nYU5VwMDm67tjJYHi3jnC6PuKRapsXfziC8wdLroXe7CAJxQ7): protocol: read tcp 159.223.143.39:45318->81.171.14.70:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 13:20:26.230219436 (189h52m27.057927837s) [noise=true , address=81.171.14.70:4046 , piece_id=T2PFYRKNF7IFA2275HKTEP6WBDBZKL56BZ26H7IHDNMWWRX56JJQ, seg_pos={0 0}, upload=197386]: Long tail cancellation due to optimal threshold Dec 13 13:20:26.230249796 (189h52m27.057958176s) [seg_pos={0 0}, upload=197386]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 13 13:20:26.230536147 (189h52m27.05824454s) [upload=197386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 13:20:26.230543508 (189h52m27.058251899s) [upload=197386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 13:20:26.379840216 (189h52m27.207548608s) [upload=197386]: Done uploading segments... Dec 13 13:20:26.379861196 (189h52m27.207569587s) [upload=197386]: Upload finished. err: <nil> Duration: 3.891045808s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)