Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 221.067111ms FAIL (2025-10-26 18:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 221.09573ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 287.143147ms FAIL (2025-10-26 18:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 287.201504ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 416.177058ms FAIL (2025-10-26 18:10:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 416.204861ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 452.706033ms FAIL (2025-10-26 18:09:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 452.734174ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 537.95641ms FAIL (2025-10-26 18:09:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 538.152054ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 289.968279ms FAIL (2025-10-26 18:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 289.997567ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 242.927919ms FAIL (2025-10-26 18:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 242.961817ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 503.790829ms FAIL (2025-10-26 18:09:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 503.821379ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 217.80333ms FAIL (2025-10-26 18:08:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 217.832373ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 475.228467ms FAIL (2025-10-26 18:08:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 475.258054ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 208.885181ms FAIL (2025-10-26 18:08:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 208.917319ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 468.153861ms FAIL (2025-10-26 18:08:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 468.176096ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 287.137571ms FAIL (2025-10-26 18:08:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 287.165378ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 489.240036ms FAIL (2025-10-26 18:08:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 489.270179ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 329.769026ms FAIL (2025-10-26 18:08:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 329.79273ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 214.259362ms FAIL (2025-10-26 18:07:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 214.287591ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 437.195626ms FAIL (2025-10-26 18:07:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 437.228126ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 241.302232ms FAIL (2025-10-26 18:07:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 241.341467ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 311.874434ms FAIL (2025-10-26 18:07:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 311.913149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 525.426064ms FAIL (2025-10-26 18:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 525.467528ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 232.375981ms FAIL (2025-10-26 18:07:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 232.413306ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 322.422049ms FAIL (2025-10-26 18:06:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 322.450686ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 407.851605ms FAIL (2025-10-26 18:06:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 407.882345ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 240.242664ms FAIL (2025-10-26 18:06:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 240.277785ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 406.485566ms FAIL (2025-10-26 18:06:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 406.523064ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 232.132824ms FAIL (2025-10-26 18:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 232.176431ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 408.226541ms FAIL (2025-10-26 18:06:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 408.271304ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 243.674118ms FAIL (2025-10-26 18:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 243.708287ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 516.052205ms FAIL (2025-10-26 18:05:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 516.08361ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 239.226918ms FAIL (2025-10-26 18:05:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 239.256637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 259.427365ms FAIL (2025-10-26 18:05:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 259.463648ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 410.761916ms FAIL (2025-10-26 18:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 410.806385ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 225.827503ms FAIL (2025-10-26 18:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 225.873652ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 362.522866ms FAIL (2025-10-26 18:05:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 362.555761ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 227.475544ms FAIL (2025-10-26 18:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 227.501715ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 407.21925ms FAIL (2025-10-26 18:04:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 407.252378ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 250.966759ms FAIL (2025-10-26 18:04:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 250.994677ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 454.037399ms FAIL (2025-10-26 18:04:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 454.072051ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 421.538789ms FAIL (2025-10-26 18:04:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 421.576566ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 207.061602ms FAIL (2025-10-26 18:03:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 207.097482ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 318.71603ms FAIL (2025-10-26 18:03:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 318.743927ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 404.318203ms FAIL (2025-10-26 18:03:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 404.347079ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 487.564604ms FAIL (2025-10-26 18:03:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 487.593933ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 237.687057ms FAIL (2025-10-26 18:03:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 237.79232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 373.230994ms FAIL (2025-10-26 18:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 373.276516ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 261.770799ms FAIL (2025-10-26 18:03:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 261.814202ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 407.469927ms FAIL (2025-10-26 18:02:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 407.499695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 238.108017ms FAIL (2025-10-26 18:02:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 238.144479ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 409.547433ms FAIL (2025-10-26 18:02:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 409.579249ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 468.035996ms FAIL (2025-10-26 18:02:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 468.068924ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)