Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 385.856687ms FAIL (2025-10-26 17:43:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 385.90452ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 537.866896ms FAIL (2025-10-26 17:43:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 537.911973ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 265.297575ms FAIL (2025-10-26 17:43:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 265.419801ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 419.370451ms FAIL (2025-10-26 17:43:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 419.407523ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 286.222076ms FAIL (2025-10-26 17:43:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 286.250521ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 393.091708ms FAIL (2025-10-26 17:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 393.123421ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 238.523896ms FAIL (2025-10-26 17:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 238.551582ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 677.961398ms FAIL (2025-10-26 17:42:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 678.000379ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 447.309702ms FAIL (2025-10-26 17:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 447.34039ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 225.694244ms FAIL (2025-10-26 17:42:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 225.724803ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 306.860789ms FAIL (2025-10-26 17:42:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 306.896148ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 512.728666ms FAIL (2025-10-26 17:41:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 512.761949ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 216.219159ms FAIL (2025-10-26 17:41:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 216.246344ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 534.072746ms FAIL (2025-10-26 17:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 534.299477ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.002969553s FAIL (2025-10-26 17:41:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 30.002999373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 228.248572ms FAIL (2025-10-26 17:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 228.27913ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 308.497709ms FAIL (2025-10-26 17:41:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 308.533317ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 454.631249ms FAIL (2025-10-26 17:41:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 454.665891ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 393.765008ms FAIL (2025-10-26 17:40:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 393.804603ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 225.800078ms FAIL (2025-10-26 17:40:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 225.82822ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.022753999s FAIL (2025-10-26 17:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:39:50.491359277 (714h21m36.401731132s) [upload=742427]: Starting upload Oct 26 17:39:50.491427492 (714h21m36.401799361s) [upload=742427]: Done starting upload Oct 26 17:39:50.491502872 (714h21m36.401874743s) [upload=742427]: Uploading segments... Oct 26 17:39:50.491684088 (714h21m36.402055947s) [upload=742427]: Got next segment. Inline: false Oct 26 17:39:50.491735422 (714h21m36.402107291s) [upload=742427]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:39:50.491740693 (714h21m36.402112561s) [upload=742427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:39:50.521756086 (714h21m36.432127957s) [seg_pos={0 0}, upload=742427]: Begin upload segment... Oct 26 17:39:50.521846885 (714h21m36.432218757s) [seg_pos={0 0}, upload=742427]: Done begin upload segment. Oct 26 17:39:50.521994942 (714h21m36.432366813s) [noise=true , address=145.79.227.20:4438 , piece_id=ITTN6LQIG3BJW6WASYJRMGMK5YVVCUI2I23DZLK4ZBEASTWLERXA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.522125962 (714h21m36.432497833s) [noise=true , address=77.90.31.21:4879 , piece_id=NQGZ2A542GLV5A2BI7EFIGPUF7QLZZDVJQOCQOFQU5WYEIGWOUXA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.523156539 (714h21m36.433528424s) [noise=true , address=145.79.227.11:4731 , piece_id=M54ATFB7AE4LKZUKOW2QAGTCX3NAMILJRIXLDTVPNGRWIRXIFZ4A, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.524080595 (714h21m36.434452467s) [noise=true , address=77.90.31.19:4843 , piece_id=VIVN3DWM7VMWKCIX5TGGT7J343Z4HLDXSNSOPSW2NDVJLCI6F3WQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.524505019 (714h21m36.434876893s) [noise=true , address=77.90.31.11:4555 , piece_id=FYNCS25QT5NDU5TF2DT5SFATTWXSZF3RV47R7KXGRW5QP3DV7STA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.524853746 (714h21m36.435225618s) [noise=true , address=145.79.227.8:4329 , piece_id=O2TRLVTYVYG5AFZNCSRJC2DIVZWDZZEAYII2DZ3AN26YZPZCFU6A, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.525151088 (714h21m36.435522961s) [noise=true , address=145.79.227.14:4839 , piece_id=PNRQKOLFRMH3MA4GBLWM3EEWDFPIUYNO3HZ47XGS34FRVJJ3UWEQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.525609201 (714h21m36.435981079s) [noise=true , address=145.79.227.7:4295 , piece_id=OFNQ3F4NYNE6323SEQLP4MCKT77QIIFWHF53YU76NLMOYOCPQG7Q, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.526273052 (714h21m36.436644927s) [noise=true , address=145.79.227.24:4583 , piece_id=UKCEB7UEE5YN3LPS5MME2YL7XBIKC5FOX2ZHMOGKCYA3DM5YQSTA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.526952858 (714h21m36.437324738s) [noise=true , address=45.91.116.5:4299 , piece_id=MPFG7FK2MHN3BCZCNNE6BYKKTDJBDP6JTGWX73ITW3EK5SJA6LWQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.527249921 (714h21m36.437621799s) [noise=true , address=77.90.31.15:4699 , piece_id=M53QRHVXKVUGGL22B75ZXHXD2FEGIJJKIYQQPVIIXSFAQIVPFWMQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.527615683 (714h21m36.437987559s) [noise=true , address=145.79.227.14:4840 , piece_id=GNLYEOKHS6F2FHHZQEUSBN7CZB2SFHJIPXR3OMK7WWBUI76EQBQQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.527973944 (714h21m36.438345812s) [noise=true , address=145.79.227.19:4077 , piece_id=HVWXTV7KXLD5PVAAVF7YW2ECZOYRVX27KKREUASTSHTERTTJMHHQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.528268188 (714h21m36.438640056s) [noise=true , address=145.79.227.21:4475 , piece_id=Z4Z5KHPCZXCWVRJAHWQHKNLL2F7GO6VUOCRJYVXOZ7GDM34Q6YCA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.528662574 (714h21m36.439034431s) [noise=true , address=145.79.227.6:4258 , piece_id=CHMTETGHDEGGGQP2GIGFNZL2EPMMMQFGRKCDWXGQPONNYAN4UBGA, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.529276800 (714h21m36.439648659s) [noise=true , address=145.79.227.18:4007 , piece_id=HOF4O2IFTALVE57MPUN2MBWGLMVCYNRHM6BCCLH242DT3UBYXJKQ, seg_pos={0 0}, upload=742427]: Uploading piece... Oct 26 17:39:50.595356670 (714h21m36.505728528s) [upload=742427]: Next returned nil segment Oct 26 17:39:50.595372673 (714h21m36.505744541s) [upload=742427]: Waiting for error group managing segments... Oct 26 17:39:50.598901654 (714h21m36.509273512s) [noise=true , address=77.90.31.19:4843 , piece_id=VIVN3DWM7VMWKCIX5TGGT7J343Z4HLDXSNSOPSW2NDVJLCI6F3WQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.601250889 (714h21m36.511622763s) [noise=true , address=145.79.227.20:4438 , piece_id=ITTN6LQIG3BJW6WASYJRMGMK5YVVCUI2I23DZLK4ZBEASTWLERXA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.601279825 (714h21m36.511651683s) [noise=true , address=77.90.31.21:4879 , piece_id=NQGZ2A542GLV5A2BI7EFIGPUF7QLZZDVJQOCQOFQU5WYEIGWOUXA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.602718892 (714h21m36.51309075s) [noise=true , address=77.90.31.15:4699 , piece_id=M53QRHVXKVUGGL22B75ZXHXD2FEGIJJKIYQQPVIIXSFAQIVPFWMQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.603459702 (714h21m36.513831562s) [noise=true , address=145.79.227.18:4007 , piece_id=HOF4O2IFTALVE57MPUN2MBWGLMVCYNRHM6BCCLH242DT3UBYXJKQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.603741081 (714h21m36.514112939s) [noise=true , address=145.79.227.7:4295 , piece_id=OFNQ3F4NYNE6323SEQLP4MCKT77QIIFWHF53YU76NLMOYOCPQG7Q, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.604010575 (714h21m36.514382438s) [noise=true , address=145.79.227.8:4329 , piece_id=O2TRLVTYVYG5AFZNCSRJC2DIVZWDZZEAYII2DZ3AN26YZPZCFU6A, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.604194286 (714h21m36.514566144s) [noise=true , address=77.90.31.11:4555 , piece_id=FYNCS25QT5NDU5TF2DT5SFATTWXSZF3RV47R7KXGRW5QP3DV7STA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.604578381 (714h21m36.514950243s) [noise=true , address=145.79.227.21:4475 , piece_id=Z4Z5KHPCZXCWVRJAHWQHKNLL2F7GO6VUOCRJYVXOZ7GDM34Q6YCA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.604593636 (714h21m36.514965496s) [noise=true , address=145.79.227.6:4258 , piece_id=CHMTETGHDEGGGQP2GIGFNZL2EPMMMQFGRKCDWXGQPONNYAN4UBGA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.605118141 (714h21m36.515489996s) [noise=true , address=145.79.227.11:4731 , piece_id=M54ATFB7AE4LKZUKOW2QAGTCX3NAMILJRIXLDTVPNGRWIRXIFZ4A, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.605793577 (714h21m36.516165437s) [noise=true , address=145.79.227.14:4840 , piece_id=GNLYEOKHS6F2FHHZQEUSBN7CZB2SFHJIPXR3OMK7WWBUI76EQBQQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.606195085 (714h21m36.516566943s) [noise=true , address=145.79.227.19:4077 , piece_id=HVWXTV7KXLD5PVAAVF7YW2ECZOYRVX27KKREUASTSHTERTTJMHHQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.606204160 (714h21m36.516576021s) [seg_pos={0 0}, upload=742427]: Segment reached optimal threshold of 13 pieces. Oct 26 17:39:50.606223980 (714h21m36.516595834s) [noise=true , address=145.79.227.24:4583 , piece_id=UKCEB7UEE5YN3LPS5MME2YL7XBIKC5FOX2ZHMOGKCYA3DM5YQSTA, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.606358425 (714h21m36.516730282s) [noise=true , address=145.79.227.14:4839 , piece_id=PNRQKOLFRMH3MA4GBLWM3EEWDFPIUYNO3HZ47XGS34FRVJJ3UWEQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Oct 26 17:39:50.606368550 (714h21m36.516740421s) [noise=true , address=145.79.227.14:4839 , piece_id=PNRQKOLFRMH3MA4GBLWM3EEWDFPIUYNO3HZ47XGS34FRVJJ3UWEQ, seg_pos={0 0}, upload=742427]: Long tail cancellation due to optimal threshold Oct 26 17:39:50.606523424 (714h21m36.516895279s) [noise=true , address=45.91.116.5:4299 , piece_id=MPFG7FK2MHN3BCZCNNE6BYKKTDJBDP6JTGWX73ITW3EK5SJA6LWQ, seg_pos={0 0}, upload=742427]: Done uploading piece. err: <nil> Oct 26 17:39:50.606545105 (714h21m36.516916972s) [seg_pos={0 0}, upload=742427]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Oct 26 17:39:50.606606796 (714h21m36.51697865s) [upload=742427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:39:50.606612053 (714h21m36.516983924s) [upload=742427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:40:20.478609734 (714h22m6.388981609s) [upload=742427]: Done uploading segments... Oct 26 17:40:20.478639058 (714h22m6.389010927s) [upload=742427]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.022788414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 318.660971ms FAIL (2025-10-26 17:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 318.689541ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 223.620644ms FAIL (2025-10-26 17:40:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 223.702446ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 448.352504ms FAIL (2025-10-26 17:39:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 448.398168ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 237.75958ms FAIL (2025-10-26 17:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 237.794883ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 433.532014ms FAIL (2025-10-26 17:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 433.566374ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 239.405422ms FAIL (2025-10-26 17:39:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 239.435309ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 208.897169ms FAIL (2025-10-26 17:39:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 209.454192ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 244.015039ms FAIL (2025-10-26 17:38:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 244.048148ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 254.250605ms FAIL (2025-10-26 17:38:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 254.284601ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 511.709096ms FAIL (2025-10-26 17:38:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 511.748694ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 271.995385ms FAIL (2025-10-26 17:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 272.026726ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 287.145553ms FAIL (2025-10-26 17:37:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 287.193978ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 412.056061ms FAIL (2025-10-26 17:37:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 412.095992ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 248.036563ms FAIL (2025-10-26 17:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 248.057693ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 246.204167ms FAIL (2025-10-26 17:36:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 246.226077ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 228.204028ms FAIL (2025-10-26 17:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 228.226283ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000879569s FAIL (2025-10-26 17:36:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:35:10.999213649 (714h16m56.909585512s) [upload=742343]: Starting upload Oct 26 17:35:10.999308537 (714h16m56.909680406s) [upload=742343]: Done starting upload Oct 26 17:35:10.999454044 (714h16m56.909825904s) [upload=742343]: Uploading segments... Oct 26 17:35:10.999515546 (714h16m56.909887415s) [upload=742343]: Got next segment. Inline: true Oct 26 17:35:10.999521345 (714h16m56.909893213s) [upload=742343]: Waiting for error group managing segments... Oct 26 17:35:10.999568776 (714h16m56.909940636s) [upload=742343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:35:10.999579132 (714h16m56.909951s) [upload=742343]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:35:10.999582744 (714h16m56.909954606s) [upload=742343]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:10.967370059 (714h17m56.87774192s) [upload=742343]: Done uploading segments... Oct 26 17:36:10.967407380 (714h17m56.877779251s) [upload=742343]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000897015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 477.486517ms FAIL (2025-10-26 01:15:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.521094ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 439.021327ms FAIL (2025-10-26 01:15:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.051627ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 422.176195ms FAIL (2025-10-26 01:14:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.20207ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 509.872143ms FAIL (2025-10-26 01:13:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.89284ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 526.915374ms FAIL (2025-10-26 01:13:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 526.999681ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 462.529875ms FAIL (2025-10-26 01:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 462.569952ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.084955294s FAIL (2025-10-26 01:11:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.08498084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 531.699991ms FAIL (2025-10-26 01:10:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 531.721469ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 477.912839ms FAIL (2025-10-26 01:09:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.94853ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 457.280895ms FAIL (2025-10-26 01:08:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.311936ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 481.867316ms FAIL (2025-10-26 01:08:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 481.900338ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 845.119685ms FAIL (2025-10-26 01:07:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 845.801746ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)