Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 323.856716ms FAIL (2025-10-26 17:53:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 323.888077ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 487.367183ms FAIL (2025-10-26 17:52:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 487.400433ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 437.697263ms FAIL (2025-10-26 17:52:33 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 437.726753ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 345.869952ms FAIL (2025-10-26 17:52:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 345.899941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 215.536737ms FAIL (2025-10-26 17:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 215.595417ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 473.652998ms FAIL (2025-10-26 17:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 473.687711ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 229.130579ms FAIL (2025-10-26 17:51:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 229.183321ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 292.718849ms FAIL (2025-10-26 17:51:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 292.798438ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 389.895934ms FAIL (2025-10-26 17:51:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 390.144942ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 488.558009ms FAIL (2025-10-26 17:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 488.590255ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 468.24731ms FAIL (2025-10-26 17:51:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 468.292496ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 236.594166ms FAIL (2025-10-26 17:51:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 236.624019ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 494.865313ms FAIL (2025-10-26 17:50:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 494.896091ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 213.639129ms FAIL (2025-10-26 17:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 213.668151ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 425.96913ms FAIL (2025-10-26 17:50:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 425.998945ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 552.934461ms FAIL (2025-10-26 17:50:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 552.970276ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 234.227048ms FAIL (2025-10-26 17:49:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 234.260674ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 600.75374ms FAIL (2025-10-26 17:49:46 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 600.789951ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 453.147347ms FAIL (2025-10-26 17:49:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 453.191095ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 379.12304ms FAIL (2025-10-26 17:49:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 379.218032ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 230.251611ms FAIL (2025-10-26 17:49:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 230.277806ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 494.41277ms FAIL (2025-10-26 17:49:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 494.450456ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 218.443014ms FAIL (2025-10-26 17:48:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 218.47563ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 402.706854ms FAIL (2025-10-26 17:48:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 402.740047ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 398.059062ms FAIL (2025-10-26 17:48:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 398.091936ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 420.786069ms FAIL (2025-10-26 17:48:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 420.818149ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 228.193426ms FAIL (2025-10-26 17:48:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 228.220518ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 226.279887ms FAIL (2025-10-26 17:47:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 226.402178ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 458.353116ms FAIL (2025-10-26 17:47:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 458.38947ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 495.231883ms FAIL (2025-10-26 17:47:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 495.274149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 287.098888ms FAIL (2025-10-26 17:47:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 287.134283ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 469.674175ms FAIL (2025-10-26 17:47:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 469.705258ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 396.382972ms FAIL (2025-10-26 17:47:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 396.434272ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 305.202255ms FAIL (2025-10-26 17:46:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 305.235287ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 443.307368ms FAIL (2025-10-26 17:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 443.341942ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 273.883272ms FAIL (2025-10-26 17:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 273.947628ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 273.961015ms FAIL (2025-10-26 17:46:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 273.993572ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 557.816952ms FAIL (2025-10-26 17:46:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 557.844547ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 282.068481ms FAIL (2025-10-26 17:46:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 282.107101ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 417.397346ms FAIL (2025-10-26 17:46:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 417.426947ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 360.897922ms FAIL (2025-10-26 17:45:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 360.928561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 228.427582ms FAIL (2025-10-26 17:45:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 228.491183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 409.911031ms FAIL (2025-10-26 17:45:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 409.97696ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 315.435077ms FAIL (2025-10-26 17:45:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 315.490219ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 545.414677ms FAIL (2025-10-26 17:45:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:41:04.900459331 (714h22m50.810831206s) [upload=742446]: Starting upload Oct 26 17:41:04.912836798 (714h22m50.823208676s) [upload=742446]: Done starting upload Oct 26 17:41:04.913257889 (714h22m50.823629748s) [upload=742446]: Uploading segments... Oct 26 17:41:04.913350981 (714h22m50.82372296s) [upload=742446]: Got next segment. Inline: false Oct 26 17:41:04.913411696 (714h22m50.823783571s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:41:04.913419763 (714h22m50.823791738s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 17:41:04.941403906 (714h22m50.851775773s) [seg_pos={0 0}, upload=742446]: Begin upload segment... Oct 26 17:41:04.941478631 (714h22m50.85185049s) [seg_pos={0 0}, upload=742446]: Done begin upload segment. Oct 26 17:41:04.941505262 (714h22m50.851877132s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.941826128 (714h22m50.852197996s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.944116755 (714h22m50.85448861s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.945015122 (714h22m50.855386992s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.946103563 (714h22m50.856475423s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.947971760 (714h22m50.858343616s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.949935958 (714h22m50.860307819s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.954237069 (714h22m50.864608927s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.958180089 (714h22m50.868551952s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.959310247 (714h22m50.869682108s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.960590839 (714h22m50.8709627s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.963706957 (714h22m50.874078819s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.964963546 (714h22m50.875335408s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.967546817 (714h22m50.877918678s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.969842775 (714h22m50.880214637s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:04.972912720 (714h22m50.883284592s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Uploading piece... Oct 26 17:41:05.030692599 (714h22m50.941064467s) [upload=742446]: Next returned nil segment Oct 26 17:41:05.030712495 (714h22m50.941084354s) [upload=742446]: Waiting for error group managing segments... Oct 26 17:41:05.040229565 (714h22m50.950601427s) [noise=true , address=145.79.227.13:4801 , piece_id=KYKBM62L5LJH4ZPKALURVK3EXAPOLWXA4VVEDZ245PYAQFWTUNDQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.040940105 (714h22m50.951311974s) [noise=true , address=145.79.227.6:4258 , piece_id=7QCEHOCHYLS3NXDIBVZLOLMVZIVZDMUQFVQT7EX2U2AES4RXNPLQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.041892214 (714h22m50.952264075s) [noise=true , address=145.79.227.20:4438 , piece_id=YOCUT6PX2SX63VUOMB3BKYYFGLMF3GDTRKOYVRQE33SSMHIL7V6A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043323654 (714h22m50.953695516s) [noise=true , address=45.91.116.7:4009 , piece_id=2DWBINXQTDY5QU3BMFLUC5DZZTCOTUMERK6ESVHZYND6VD7CLVAA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.043666299 (714h22m50.954038174s) [noise=true , address=145.79.227.25:4619 , piece_id=IISVDJ5EIUQP2V5NYDOEO7YPZKDEMA7MSJPLTTTXMEBWEQDU7CBQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044225145 (714h22m50.954597001s) [noise=true , address=145.79.227.18:4007 , piece_id=4SUZKDM2TXNKBS2ERL7CWRPCQNWRPJK6WQAY2XTWP3RR3CR5QWTA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.044791220 (714h22m50.955163086s) [noise=true , address=145.79.227.14:4838 , piece_id=LEQU62QIVIDT53LNFOZNU4MYVEPAEO4KP67A3CB2HL3EFEGNPPUQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045308574 (714h22m50.955680432s) [noise=true , address=145.79.227.9:4367 , piece_id=C4CRJ6HPOHPWFGIRFDGFW6LSRQEZ5G7EEHIRRO6JCDLHKMW2CPSQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045404363 (714h22m50.955776223s) [noise=true , address=145.79.227.8:4329 , piece_id=EAREQKM5FIGBTAOIIXEYUTTKQ3X4Q2GIGPFP7POEWC7SF3IPWFXQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045666231 (714h22m50.956038091s) [noise=true , address=145.79.227.21:4475 , piece_id=ZKTUNJPUA32E6SJFLQ75UFGK5Q7IX6XWBVTHOQA72UZTL43CP65Q, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.045909859 (714h22m50.956281729s) [noise=true , address=145.79.227.17:4971 , piece_id=QONKMJRWQZVVXW4GQ644FEMUYN5JNUNFNNOHBDPJP6SVRSRYHIWA, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047649944 (714h22m50.958021813s) [noise=true , address=77.90.31.21:4879 , piece_id=CHSCJRZKOZZDF5ETDUBB55VBMF7JLRS55YEEWR4NFDM56K3FYOAQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047675094 (714h22m50.958046953s) [noise=true , address=45.91.116.11:4118 , piece_id=QY7BZAHHWBZNV7RNTLSBYMFJGXQEEGYLCB67U7CVJ5UXAM5UXE5A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: <nil> Oct 26 17:41:05.047802715 (714h22m50.958174574s) [seg_pos={0 0}, upload=742446]: Segment reached optimal threshold of 13 pieces. Oct 26 17:41:05.047963600 (714h22m50.958335456s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.047982520 (714h22m50.958354395s) [noise=true , address=145.79.227.3:4151 , piece_id=MJOFYU7DT5UKNSGEFXEO7LOVEMNAIJVPI3I3AABDQ6CT7YOHXH4A, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048062123 (714h22m50.958433984s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048069503 (714h22m50.958441379s) [noise=true , address=145.79.227.7:4294 , piece_id=GN32Y25TLZBXO2BNZ5NQLY64EEDOBDRJR2IWZ44YSF7OMD7RCUMQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048150359 (714h22m50.958522217s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk): protocol: expected piece hash; context canceled; EOF Oct 26 17:41:05.048162128 (714h22m50.958533998s) [noise=true , address=145.79.227.21:4473 , piece_id=HHJLPLDHUULBXLSAWV32YYLR34TGZMDHZHXIZFLF3PMJV5HUVFKQ, seg_pos={0 0}, upload=742446]: Long tail cancellation due to optimal threshold Oct 26 17:41:05.048182297 (714h22m50.958554158s) [seg_pos={0 0}, upload=742446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 17:41:05.048291188 (714h22m50.958663056s) [upload=742446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:41:05.048297038 (714h22m50.958668906s) [upload=742446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:41:05.088157106 (714h22m50.998528979s) [upload=742446]: Done uploading segments... Oct 26 17:41:05.088182492 (714h22m50.998554351s) [upload=742446]: Upload finished. err: <nil> Duration: 545.439679ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 631.367942ms FAIL (2025-10-26 17:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 631.399355ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 259.966766ms FAIL (2025-10-26 17:44:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 259.998124ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 248.934885ms FAIL (2025-10-26 17:44:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:11.081414235 (714h17m56.991786094s) [upload=742363]: Starting upload Oct 26 17:36:11.081484435 (714h17m56.991856303s) [upload=742363]: Done starting upload Oct 26 17:36:11.081546300 (714h17m56.991918161s) [upload=742363]: Uploading segments... Oct 26 17:36:11.081592579 (714h17m56.991964449s) [upload=742363]: Got next segment. Inline: true Oct 26 17:36:11.081598125 (714h17m56.991969989s) [upload=742363]: Waiting for error group managing segments... Oct 26 17:36:11.081640440 (714h17m56.992012308s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:11.081646014 (714h17m56.992017873s) [upload=742363]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:11.081649561 (714h17m56.99202142s) [upload=742363]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:11.041444243 (714h18m56.951816106s) [upload=742363]: Done uploading segments... Oct 26 17:37:11.041463627 (714h18m56.951835495s) [upload=742363]: Upload finished. err: metaclient: context deadline exceeded Duration:* 248.971784ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 298.066194ms FAIL (2025-10-26 17:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 26 17:36:06.225517861 (714h17m52.135889719s) [upload=742361]: Starting upload Oct 26 17:36:06.225586777 (714h17m52.135958655s) [upload=742361]: Done starting upload Oct 26 17:36:06.225718883 (714h17m52.136090744s) [upload=742361]: Uploading segments... Oct 26 17:36:06.225796375 (714h17m52.136168252s) [upload=742361]: Got next segment. Inline: true Oct 26 17:36:06.225801904 (714h17m52.136173776s) [upload=742361]: Waiting for error group managing segments... Oct 26 17:36:06.225904525 (714h17m52.136276385s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 17:36:06.225912065 (714h17m52.136283924s) [upload=742361]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Oct 26 17:36:06.225929070 (714h17m52.136300939s) [upload=742361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:36:36.194305907 (714h18m22.104677769s) [upload=742361]: Done uploading segments... Oct 26 17:36:36.194327995 (714h18m22.104699863s) [upload=742361]: Upload finished. err: metaclient: context deadline exceeded Duration:* 298.109851ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 398.714842ms FAIL (2025-10-26 17:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518859569 (714h17m22.429231437s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.518926958 (714h17m22.429298832s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519371691 (714h17m22.429743562s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519474435 (714h17m22.42984631s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.519835502 (714h17m22.430207374s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520005483 (714h17m22.430377352s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520259682 (714h17m22.430631552s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520500836 (714h17m22.43087271s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.520723900 (714h17m22.431095772s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521078841 (714h17m22.431450715s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.521248212 (714h17m22.431620083s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.522634002 (714h17m22.433005872s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.524995091 (714h17m22.43536696s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.526246915 (714h17m22.436618789s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.527723851 (714h17m22.438095721s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Uploading piece... Oct 26 17:35:36.599251544 (714h17m22.509623398s) [upload=742353]: Next returned nil segment Oct 26 17:35:36.599271288 (714h17m22.509643156s) [upload=742353]: Waiting for error group managing segments... Oct 26 17:35:36.605839488 (714h17m22.51621135s) [noise=true , address=77.90.31.13:4637 , piece_id=LUSLUKXZUTECVXPEU7RQIX4PV4N3ZVEUMWP35UABKGT65CKU4TIA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.607638155 (714h17m22.518010016s) [noise=true , address=45.91.116.14:4193 , piece_id=2RJIZR6XJOXHPRQB2LTBUZXX3JK3JHKFLIH3VOR6PRH5WGTF3TPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609019413 (714h17m22.519391274s) [noise=true , address=45.91.116.8:4063 , piece_id=IRN7I3P2FEZIQ77UM6NSG3MR374TPYPDCXXPX55DXYZGZYJ665RA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.609283495 (714h17m22.51965535s) [noise=true , address=45.91.116.4:4948 , piece_id=B7X2AWZLHVYHTLX56BZWYLEICH43S5U77XTI2PPYV4D54SFSMKJQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616251735 (714h17m22.526623598s) [noise=true , address=45.91.116.7:4025 , piece_id=YPHMNI6D762Q7YR42VVAJ4PHXMU7H7MXIWVP5MZLI5EOLIPYNA6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.616693169 (714h17m22.52706503s) [noise=true , address=145.79.227.18:4017 , piece_id=U2UQZMG6GGSYFEYGUGZUJATI5ZFBQHEJTDHOT4PKKGKEDS23PWCA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.617159098 (714h17m22.527530956s) [noise=true , address=77.90.31.15:4711 , piece_id=JXZJORAWWVKT5WTGKNZVLUEY4SEM5JM3KCXDCEN6JX2J4NAVSIPA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.621209833 (714h17m22.531581691s) [noise=true , address=77.90.31.17:4792 , piece_id=R54HN23K5GRT44J6QQU4TV2LYCH653254AE56CTTU3IS3CP6XMMA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625554017 (714h17m22.535925878s) [noise=true , address=45.91.116.5:4327 , piece_id=JVPS6S2TUI646QKV47PHNKRQQFKKACBKK526L4K7HLZV2BRH5FQQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625665874 (714h17m22.536037742s) [noise=true , address=145.79.227.40:4079 , piece_id=7XWXUF7MQWI4YYTWPWTAYJDO344QSGVFES6K3GCZCXH4SEFWHHWQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.625924377 (714h17m22.536296238s) [noise=true , address=77.90.31.18:4810 , piece_id=7ECZX4IGFVZA6LTW2D7YIXTJIOZSATAJWUUHS3JDVRTBBSKWZKAQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.629367469 (714h17m22.53973933s) [noise=true , address=77.90.31.19:4867 , piece_id=KU76TU4KU5MTVGHPCXTCZ6T652QWAA46LMLIAMEFD57LFB2EEF5A, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.632031717 (714h17m22.542403577s) [noise=true , address=77.90.31.10:4548 , piece_id=22CUFYW7LIFUZTDUP7UQFO2LZ2T7YF4WD7A54UZWI6L4LSYYZRJA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633449709 (714h17m22.543821564s) [noise=true , address=145.79.227.16:4941 , piece_id=SS6V6HU6WDJ2DTFSA6ZC3K4NHZ7CZEXPSZJEQGCXIUWYSJDCUOGA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: <nil> Oct 26 17:35:36.633493523 (714h17m22.543865392s) [seg_pos={0 0}, upload=742353]: Segment reached optimal threshold of 14 pieces. Oct 26 17:35:36.633655015 (714h17m22.544026878s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633671815 (714h17m22.544043695s) [noise=true , address=145.79.227.6:4279 , piece_id=OFZGOOHUOWWHX2VBL3E423W4IKXMTEINC4HIP45ZFRD4IBKVBDYA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633805254 (714h17m22.54417711s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gC8PAH3JvFmoFrvSAF5xiBhQDMZ1ScfxuxVBrNQb5A1odWsvX): protocol: expected piece hash; context canceled; EOF Oct 26 17:35:36.633817804 (714h17m22.544189678s) [noise=true , address=45.91.116.6:4004 , piece_id=NWH3JW23OPXSJNJKSFD6LDWJAQVE3EWPDUYNO7HKPMZABX42DMRQ, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.633897227 (714h17m22.544269096s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: read tcp 159.223.143.39:34576->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 17:35:36.633907779 (714h17m22.544279658s) [noise=true , address=193.149.164.143:4702 , piece_id=GV6HY7SIRVZIPTNABKL75UM3O6FNFU4G4YQ27QA6F7TX7FQ27EWA, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634034365 (714h17m22.544406235s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:60266->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Oct 26 17:35:36.634048149 (714h17m22.544420027s) [noise=true , address=64.49.13.27:4967 , piece_id=WRHSQBWYQNYWAC3R3UFPSG5CB536E4VRVPQBJ7IZ7HZWP2LL3J6Q, seg_pos={0 0}, upload=742353]: Long tail cancellation due to optimal threshold Oct 26 17:35:36.634070344 (714h17m22.544442216s) [seg_pos={0 0}, upload=742353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 17:35:36.634196718 (714h17m22.544568592s) [upload=742353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 17:35:36.634204926 (714h17m22.544576806s) [upload=742353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 17:37:36.484087741 (714h19m22.394459609s) [upload=742353]: Done uploading segments... Oct 26 17:37:36.484103325 (714h19m22.394475193s) [upload=742353]: Upload finished. err: metaclient: context deadline exceeded Duration: 398.743019ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)