Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................X..X...X..XX..X...X..............X] duration: 38.121754964s trace: https://jaeger.dev.storj.io/trace/30625685be2c4b28 FAIL (2026-03-10 22:37:26 UTC) operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments... Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment... Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment. Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece... Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments... Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil> Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments... Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil> Duration: 38.121806762s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 4.395709462s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:37:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 4.395751435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 167.64528ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:37:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 167.664964ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...............X.XXXXXXXXXXXXXXXXXXX.XXX...XX...............] duration: 2.012938884s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:37:10 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.012988756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 239.623312ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:36:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 239.667732ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [................X.XXXXXXXXXXXXXXXXXXX.XXX...XX..............] duration: 729.626919ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:36:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 729.654847ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 13.394623472s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 13.394651996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 352.015013ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:35:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 352.103639ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X..X...X..XX..X...X..............] duration: 1.579879765s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:35:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.579929704s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 753.601517ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:35:41 UTC) orbiter warning: context deadline exceeded Duration: 753.627093ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.................X.XXXXXXXXXXXXXXXXXXX.XXX...XX.............] duration: 1.233596547s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:35:33 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.233624345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.018654711s trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:35:25 UTC) orbiter warning: context deadline exceeded Duration: 1.018688536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X..X...X..XX..X...X.............] duration: 2.309165121s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:35:20 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.309189468s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.622271723s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:35:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.622296932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..................X.XXXXXXXXXXXXXXXXXXX.XXX...XX............] duration: 762.926047ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:34:45 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 762.951835ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 241.337139ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:34:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 241.362153ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.039316925s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.039343034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X..X...X..XX..X...X............] duration: 1.332267782s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:34:32 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.33229737s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 633.289596ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:34:13 UTC) orbiter warning: context deadline exceeded Duration: 633.316625ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 248.038127ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:34:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 248.079133ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...................X.XXXXXXXXXXXXXXXXXXX.XXX...XX...........] duration: 654.147564ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:33:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 654.179512ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.637071396s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.637101661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....................X.XXXXXXXXXXXXXXXXXXX.XXX...XX..........] duration: 826.064333ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:33:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 826.093468ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 807.514171ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:33:34 UTC) orbiter warning: context deadline exceeded Duration: 807.539644ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X..X...X..XX..X...X...........] duration: 1.132945061s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:33:26 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.132963208s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 337.127634ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:33:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 337.153389ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....................X.XXXXXXXXXXXXXXXXXXX.XXX...XX.........] duration: 1.016384375s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:32:59 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.016409936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................X..X...X..XX..X...X..........] duration: 1.769314623s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:32:55 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.769341169s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.71836135s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:32:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.718391402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 794.307036ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:32:23 UTC) orbiter warning: context deadline exceeded Duration: 794.331131ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 218.373645ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:32:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 218.400948ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 914.894108ms trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 914.980039ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................X..X...X..XX..X...X.........] duration: 1.378430663s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:32:07 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.378445474s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 235.115333ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:31:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 235.165881ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.74151768s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.741550497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [......................X.XXXXXXXXXXXXXXXXXXX.XXX...XX........] duration: 707.278076ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:31:48 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 707.307503ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 612.957036ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:31:35 UTC) orbiter warning: context deadline exceeded Duration: 612.977833ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 743.537344ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:31:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 743.563896ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.750164769s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:31:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 2.750205546s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.638764343s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:31:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.638794528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 958.900186ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:31:14 UTC) orbiter warning: context deadline exceeded Duration: 958.925755ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................X..X...X..XX..X...X........] duration: 1.589233083s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:30:56 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.589262383s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.......................X.XXXXXXXXXXXXXXXXXXX.XXX...XX.......] duration: 824.225995ms trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:30:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 824.257093ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.279316336s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.279349026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 224.947906ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:30:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 224.97816ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 843.034238ms trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:30:10 UTC) orbiter warning: context deadline exceeded Duration: 843.077751ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002582473s trace: https://jaeger.dev.storj.io/trace/326c0a726d051422 FAIL (2026-03-10 22:30:09 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00259676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.199117977s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:30:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 2.19918704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................X..X...X..XX..X...X.......] duration: 2.136580103s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:30:04 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.136611416s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [........................X.XXXXXXXXXXXXXXXXXXX.XXX...XX......] duration: 1.114926532s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:29:39 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.114976491s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)