Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................X..X...X..XX..X...X......] duration: 1.810655688s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:29:32 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.810681402s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 428.07697ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:29:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 428.106461ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.42816708s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 1.428196087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................X..X...X..XX..X...X.....] duration: 1.296467947s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:28:55 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.296493532s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.179342819s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:28:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 1.179439525s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.........................X.XXXXXXXXXXXXXXXXXXX.XXX...XX.....] duration: 1.14834966s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:28:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.148379178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.590072971s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 5.590567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................X..X...X..XX..X...X....] duration: 3.230883367s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:27:55 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.230931245s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 10.000933319s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 10.000955623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..........................X.XXXXXXXXXXXXXXXXXXX.XXX...XX....] duration: 1.047538336s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:27:38 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.047573947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...........................X.XXXXXXXXXXXXXXXXXXX.XXX...XX...] duration: 2.729222414s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:27:20 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.729242449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................................X..X...X..XX..X...X...] duration: 1.864987546s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:27:19 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.865008372s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 32.898912722s trace: https://jaeger.dev.storj.io/trace/58132c67a664c204 FAIL (2026-03-10 22:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece... Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments... Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil> Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments... Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil> Duration: 32.898962277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 318.852759ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:27:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 318.879ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................................X..X...X..XX..X...X..] duration: 4.233330983s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:27:05 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.233363794s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [............................X.XXXXXXXXXXXXXXXXXXX.XXX...XX..] duration: 1.134365516s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:26:53 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.134386943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.255096064s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:26:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 1.255117016s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................................X..X...X..XX..X...X.] duration: 2.790200349s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:26:37 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.79023091s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................................X..X...X..XX..X...X] duration: 1m55.774936374s trace: https://jaeger.dev.storj.io/trace/217fe090e1f689dd FAIL (2026-03-10 22:26:34 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:24:38.573566609 (107h24m36.170337632s) [upload=120230]: Starting upload Mar 10 22:24:38.573653641 (107h24m36.170424673s) [upload=120230]: Done starting upload Mar 10 22:24:38.573951421 (107h24m36.170722438s) [upload=120230]: Uploading segments... Mar 10 22:24:38.574359305 (107h24m36.171130328s) [upload=120230]: Got next segment. Inline: false Mar 10 22:24:38.574409409 (107h24m36.171180435s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:24:38.574415031 (107h24m36.171186055s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:24:38.604757762 (107h24m36.201528785s) [seg_pos={0 0}, upload=120230]: Begin upload segment... Mar 10 22:24:38.604830622 (107h24m36.201601634s) [seg_pos={0 0}, upload=120230]: Done begin upload segment. Mar 10 22:24:38.604894111 (107h24m36.201665137s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605478475 (107h24m36.202249508s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.605951116 (107h24m36.202722138s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606150364 (107h24m36.202921392s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606418550 (107h24m36.203189576s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606579453 (107h24m36.203350513s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606777909 (107h24m36.203548931s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.606998332 (107h24m36.203769355s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607225639 (107h24m36.203996662s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607430881 (107h24m36.204201905s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607642079 (107h24m36.204413103s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.607884327 (107h24m36.204655354s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608146647 (107h24m36.204917679s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608306309 (107h24m36.205077342s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608582544 (107h24m36.20535357s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.608778310 (107h24m36.205549357s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609019146 (107h24m36.20579017s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.609140233 (107h24m36.205911256s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Uploading piece... Mar 10 22:24:38.679013052 (107h24m36.275784066s) [upload=120230]: Next returned nil segment Mar 10 22:24:38.679054380 (107h24m36.275825422s) [upload=120230]: Waiting for error group managing segments... Mar 10 22:24:38.699796463 (107h24m36.296567485s) [noise=true , address=45.91.116.7:4023 , piece_id=2QV5JB24IC7YU3JOB24ULKJ4KDZJJORZ6BKRAYZO5A62UONBGUJA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.700662079 (107h24m36.297433109s) [noise=true , address=77.90.31.19:4863 , piece_id=7U5IJH7OEVZKHBIQZIUDUD4VZPKGYVVEXN46GRJAAB46STIS6X4A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.702685800 (107h24m36.299456825s) [noise=true , address=145.79.227.32:4859 , piece_id=IRGBX7OSTN4IPTQB7GL5XO4NHCVS3CEEK2LMWRCRVF3QN2DJT3CA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.707529749 (107h24m36.304300772s) [noise=true , address=79.127.219.38:4851 , piece_id=MHI4FCANK7QSTEHQA6BHFEPK7EJJKLOAGIUX5VYZWU3FKQGWSNFQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.721772771 (107h24m36.318543795s) [noise=true , address=77.90.31.16:4753 , piece_id=4NKXPUC2WNCUMZ3NRDLASLCCI6F72MOETCS6DB5OXI7NOBR7JYIA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.732228762 (107h24m36.32899979s) [noise=true , address=145.79.227.29:4963 , piece_id=ZDFPOWB4ELMEBSATI3LVJQUIS5MWVZAF4JHLL7PNMTH5JJGEFRRQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.739615136 (107h24m36.33638616s) [noise=true , address=145.79.227.26:4662 , piece_id=6ESE3AT4NNTI6BHFKWGS7H2YA2XFSIRODVGJQ7PCAKDTDXNLNYUA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.748228764 (107h24m36.344999775s) [noise=true , address=77.90.31.13:4658 , piece_id=7LXM56IOC3WZ7YLZETJDYQD32YSP4HVMEB4FPT43OJEV7BSY6ITQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:38.800956526 (107h24m36.39772755s) [noise=true , address=145.79.227.18:4013 , piece_id=QRAFJBIPJF52CIRWNQM47XXU32W7XPNWOGYJKQPQNVNTSQONLCLQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.177746065 (107h24m36.774517091s) [noise=true , address=79.127.205.234:4207 , piece_id=JNB6FJSTFWOJXLUEQ67ETEF77XNW7RKPHMJ5W7SOITPRIPOUCF5A, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.179254506 (107h24m36.776025533s) [noise=true , address=79.127.205.237:4221 , piece_id=SCTJXF4JJ4GMK7MDELBS52K52ZURQDMR6DUIPHJXKIPTU6KUNUWA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.192599836 (107h24m36.789370863s) [noise=true , address=193.149.164.70:4114 , piece_id=CPLCPUENSQVP525BY5K5EYTJ3Q54IMEVHRFLXMLK5ZPPAHHI5XYA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.194197563 (107h24m36.790968587s) [noise=true , address=193.149.164.67:4999 , piece_id=RIVZCWI6RA4X2I3APK475NGLNB3NAVQSVRK55TZI7JHWBPZC6ZKA, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198278447 (107h24m36.795049469s) [noise=true , address=193.149.164.69:4092 , piece_id=PMOANGOT4DBAKBCNID3ZQ2U6LCP4CQ3YSUKRUMLC3PSXJLZARS2Q, seg_pos={0 0}, upload=120230]: Done uploading piece. err: <nil> Mar 10 22:24:39.198302950 (107h24m36.795073975s) [seg_pos={0 0}, upload=120230]: Satisfied cohort requirements after 14 pieces Mar 10 22:24:39.198672095 (107h24m36.795443124s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13TLh8dDVr5jVRzsM7McF6pzo5Ytznyn9qLaaFsi7v6HYfVytA): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198683227 (107h24m36.79545425s) [noise=true , address=64.49.13.170:4723 , piece_id=AEVZVQZRB4YXJF6ZZSRBWSZ2QYJJRYMQZUC4DA3CNOA7FRWRXOKQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.198839141 (107h24m36.795610175s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gsNGvM8oKL9Fw1KLTrtpANYVozX2KGDDtL5BUmDhL9GDeFGof): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.198848316 (107h24m36.795619345s) [noise=true , address=45.91.116.5:4319 , piece_id=VCVR6ZJJGMAWF6SMJRDT6XTKR7O4TXFWQFYJSJHEM5IBSGHX77ZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199011046 (107h24m36.795782075s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZzQb7FEr5L5BB67RAdNniykDfARj9u3ZAQwyLN1M7cmsUQwQc): protocol: expected piece hash; context canceled; EOF Mar 10 22:24:39.199230346 (107h24m36.796001454s) [noise=true , address=193.149.164.68:4064 , piece_id=PJEJU2WQ4Q5GPG6LTNPEZAXBF56VZSJGQZB5C7GUC7CJQA76FCZQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199486603 (107h24m36.796257635s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4): protocol: read tcp 159.223.143.39:37350->167.17.72.70:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 22:24:39.199511067 (107h24m36.796282094s) [noise=true , address=167.17.72.70:4761 , piece_id=MUTEV7BHA7HA6CMRPNPMSSA2TI33D5XS3P7LKA64UANZGEJOO6PQ, seg_pos={0 0}, upload=120230]: Long tail cancellation due to optimal threshold Mar 10 22:24:39.199543878 (107h24m36.796314894s) [seg_pos={0 0}, upload=120230]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:24:39.199635684 (107h24m36.796406707s) [upload=120230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:24:39.199642485 (107h24m36.796413509s) [upload=120230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:26:34.299804217 (107h26m31.89657523s) [upload=120230]: Done uploading segments... Mar 10 22:26:34.299890190 (107h26m31.896661222s) [upload=120230]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m55.774965201s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.............................X.XXXXXXXXXXXXXXXXXXX.XXX...XX.] duration: 2.623054719s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:26:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.623089114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..............................X.XXXXXXXXXXXXXXXXXXX.XXX...XX] duration: 5.001741408s trace: https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435 FAIL (2026-03-10 22:26:16 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001765162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.569841554s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:25:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 7.569869212s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...............................X.XXXXXXXXXXXXXXXXXXX.XXX...X] duration: 5.001602334s trace: https://jaeger.dev.storj.io/trace/147b98746db943c3 FAIL (2026-03-10 22:25:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001627859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 14.91915414s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:25:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 14.919200965s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1m0.000713063s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:24:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 1m0.000725316s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [................................X.XXXXXXXXXXXXXXXXXXX.XXX...] duration: 4.156075812s trace: https://jaeger.dev.storj.io/trace/7eb5de65f46fcf30 FAIL (2026-03-10 22:24:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.156098725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 55.027619889s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:23:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 55.027648231s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.................................X.XXXXXXXXXXXXXXXXXXX.XXX..] duration: 5.001170152s trace: https://jaeger.dev.storj.io/trace/7eb5de65f46fcf30 FAIL (2026-03-10 22:23:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001198215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................X..X...X..XX..X...] duration: 8.830302513s trace: https://jaeger.dev.storj.io/trace/38eb42f2a03eeafc FAIL (2026-03-10 22:23:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:20:13.254704710 (107h20m10.851475735s) [upload=120147]: Starting upload Mar 10 22:20:13.254766828 (107h20m10.851537845s) [upload=120147]: Done starting upload Mar 10 22:20:13.255061844 (107h20m10.851832858s) [upload=120147]: Uploading segments... Mar 10 22:20:13.255106734 (107h20m10.851877763s) [upload=120147]: Got next segment. Inline: false Mar 10 22:20:13.255154691 (107h20m10.851925714s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:20:13.255159941 (107h20m10.851930964s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:20:40.708040547 (107h20m38.304811584s) [seg_pos={0 0}, upload=120147]: Begin upload segment... Mar 10 22:20:40.708122042 (107h20m38.304893076s) [seg_pos={0 0}, upload=120147]: Done begin upload segment. Mar 10 22:20:40.708182747 (107h20m38.30495378s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708256927 (107h20m38.305027962s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708754396 (107h20m38.30552542s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709090094 (107h20m38.305861124s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709138942 (107h20m38.305909956s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709566464 (107h20m38.306337487s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709645511 (107h20m38.306416537s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709985832 (107h20m38.306756855s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710145510 (107h20m38.306916538s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710429544 (107h20m38.307200567s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710658858 (107h20m38.307429882s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710837451 (107h20m38.307608473s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711140907 (107h20m38.307911939s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711249795 (107h20m38.308020814s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711720080 (107h20m38.308491106s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711814637 (107h20m38.308585668s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712262347 (107h20m38.309033373s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712843141 (107h20m38.309614164s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.778621168 (107h20m38.375392183s) [upload=120147]: Next returned nil segment Mar 10 22:20:40.778635105 (107h20m38.375406128s) [upload=120147]: Waiting for error group managing segments... Mar 10 22:20:40.783760302 (107h20m38.380531316s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.796179808 (107h20m38.392950836s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800209480 (107h20m38.396980505s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800815184 (107h20m38.397586196s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.807172872 (107h20m38.403943895s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.813580482 (107h20m38.410351505s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.009953050 (107h20m38.606724095s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.084202987 (107h20m38.68097401s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.088186679 (107h20m38.684957707s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.260272609 (107h20m38.857043622s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.265905052 (107h20m38.862676079s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.281484647 (107h20m38.878255672s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.286010701 (107h20m38.882781724s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301451738 (107h20m38.89822276s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301503695 (107h20m38.898274738s) [seg_pos={0 0}, upload=120147]: Satisfied cohort requirements after 14 pieces Mar 10 22:20:41.301862444 (107h20m38.898633472s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NkSWfYxvndQHUnnQbHe6GPAHxgPe39gU84T5yHmrKKJSfivEd): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301873924 (107h20m38.898644949s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.301927973 (107h20m38.898699s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XfehFoLvcivSkVcGttnXVK5VCSWKCpmyPX3E6rtpVdwnUbbZ4): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301939854 (107h20m38.898710878s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302167039 (107h20m38.898938066s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127sHSxqrJhmDpMx2FWkzf5fXVSf7uYw2h9PWBMQCVvp2PWmqaJ): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302186195 (107h20m38.898957222s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302439932 (107h20m38.899210966s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oXSqFgqNNvDUaEwcHoEVUuaLukC2zaEfosyoeVLbXN81B3vVY): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302452647 (107h20m38.899223681s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302483492 (107h20m38.899254504s) [seg_pos={0 0}, upload=120147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:20:41.302796425 (107h20m38.899567451s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:20:41.302805132 (107h20m38.899576155s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:21:56.042886295 (107h21m53.639657321s) [upload=120147]: Done uploading segments... Mar 10 22:21:56.043103265 (107h21m53.639874297s) [upload=120147]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.830328178s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..................................X.XXXXXXXXXXXXXXXXXXX.XXX.] duration: 1.281003857s trace: https://jaeger.dev.storj.io/trace/7eb5de65f46fcf30 FAIL (2026-03-10 22:23:26 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.28107778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X..X...X..XX..X..] duration: 7.483885296s trace: https://jaeger.dev.storj.io/trace/38eb42f2a03eeafc FAIL (2026-03-10 22:22:37 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:20:13.254704710 (107h20m10.851475735s) [upload=120147]: Starting upload Mar 10 22:20:13.254766828 (107h20m10.851537845s) [upload=120147]: Done starting upload Mar 10 22:20:13.255061844 (107h20m10.851832858s) [upload=120147]: Uploading segments... Mar 10 22:20:13.255106734 (107h20m10.851877763s) [upload=120147]: Got next segment. Inline: false Mar 10 22:20:13.255154691 (107h20m10.851925714s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:20:13.255159941 (107h20m10.851930964s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:20:40.708040547 (107h20m38.304811584s) [seg_pos={0 0}, upload=120147]: Begin upload segment... Mar 10 22:20:40.708122042 (107h20m38.304893076s) [seg_pos={0 0}, upload=120147]: Done begin upload segment. Mar 10 22:20:40.708182747 (107h20m38.30495378s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708256927 (107h20m38.305027962s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708754396 (107h20m38.30552542s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709090094 (107h20m38.305861124s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709138942 (107h20m38.305909956s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709566464 (107h20m38.306337487s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709645511 (107h20m38.306416537s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709985832 (107h20m38.306756855s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710145510 (107h20m38.306916538s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710429544 (107h20m38.307200567s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710658858 (107h20m38.307429882s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710837451 (107h20m38.307608473s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711140907 (107h20m38.307911939s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711249795 (107h20m38.308020814s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711720080 (107h20m38.308491106s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711814637 (107h20m38.308585668s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712262347 (107h20m38.309033373s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712843141 (107h20m38.309614164s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.778621168 (107h20m38.375392183s) [upload=120147]: Next returned nil segment Mar 10 22:20:40.778635105 (107h20m38.375406128s) [upload=120147]: Waiting for error group managing segments... Mar 10 22:20:40.783760302 (107h20m38.380531316s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.796179808 (107h20m38.392950836s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800209480 (107h20m38.396980505s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800815184 (107h20m38.397586196s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.807172872 (107h20m38.403943895s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.813580482 (107h20m38.410351505s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.009953050 (107h20m38.606724095s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.084202987 (107h20m38.68097401s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.088186679 (107h20m38.684957707s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.260272609 (107h20m38.857043622s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.265905052 (107h20m38.862676079s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.281484647 (107h20m38.878255672s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.286010701 (107h20m38.882781724s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301451738 (107h20m38.89822276s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301503695 (107h20m38.898274738s) [seg_pos={0 0}, upload=120147]: Satisfied cohort requirements after 14 pieces Mar 10 22:20:41.301862444 (107h20m38.898633472s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NkSWfYxvndQHUnnQbHe6GPAHxgPe39gU84T5yHmrKKJSfivEd): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301873924 (107h20m38.898644949s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.301927973 (107h20m38.898699s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XfehFoLvcivSkVcGttnXVK5VCSWKCpmyPX3E6rtpVdwnUbbZ4): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301939854 (107h20m38.898710878s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302167039 (107h20m38.898938066s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127sHSxqrJhmDpMx2FWkzf5fXVSf7uYw2h9PWBMQCVvp2PWmqaJ): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302186195 (107h20m38.898957222s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302439932 (107h20m38.899210966s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oXSqFgqNNvDUaEwcHoEVUuaLukC2zaEfosyoeVLbXN81B3vVY): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302452647 (107h20m38.899223681s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302483492 (107h20m38.899254504s) [seg_pos={0 0}, upload=120147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:20:41.302796425 (107h20m38.899567451s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:20:41.302805132 (107h20m38.899576155s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:21:56.042886295 (107h21m53.639657321s) [upload=120147]: Done uploading segments... Mar 10 22:21:56.043103265 (107h21m53.639874297s) [upload=120147]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.483915468s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X..X...X..XX..X.] duration: 33.349542174s trace: https://jaeger.dev.storj.io/trace/38eb42f2a03eeafc FAIL (2026-03-10 22:22:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:20:13.254704710 (107h20m10.851475735s) [upload=120147]: Starting upload Mar 10 22:20:13.254766828 (107h20m10.851537845s) [upload=120147]: Done starting upload Mar 10 22:20:13.255061844 (107h20m10.851832858s) [upload=120147]: Uploading segments... Mar 10 22:20:13.255106734 (107h20m10.851877763s) [upload=120147]: Got next segment. Inline: false Mar 10 22:20:13.255154691 (107h20m10.851925714s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:20:13.255159941 (107h20m10.851930964s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:20:40.708040547 (107h20m38.304811584s) [seg_pos={0 0}, upload=120147]: Begin upload segment... Mar 10 22:20:40.708122042 (107h20m38.304893076s) [seg_pos={0 0}, upload=120147]: Done begin upload segment. Mar 10 22:20:40.708182747 (107h20m38.30495378s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708256927 (107h20m38.305027962s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708754396 (107h20m38.30552542s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709090094 (107h20m38.305861124s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709138942 (107h20m38.305909956s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709566464 (107h20m38.306337487s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709645511 (107h20m38.306416537s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709985832 (107h20m38.306756855s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710145510 (107h20m38.306916538s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710429544 (107h20m38.307200567s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710658858 (107h20m38.307429882s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710837451 (107h20m38.307608473s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711140907 (107h20m38.307911939s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711249795 (107h20m38.308020814s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711720080 (107h20m38.308491106s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711814637 (107h20m38.308585668s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712262347 (107h20m38.309033373s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712843141 (107h20m38.309614164s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.778621168 (107h20m38.375392183s) [upload=120147]: Next returned nil segment Mar 10 22:20:40.778635105 (107h20m38.375406128s) [upload=120147]: Waiting for error group managing segments... Mar 10 22:20:40.783760302 (107h20m38.380531316s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.796179808 (107h20m38.392950836s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800209480 (107h20m38.396980505s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800815184 (107h20m38.397586196s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.807172872 (107h20m38.403943895s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.813580482 (107h20m38.410351505s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.009953050 (107h20m38.606724095s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.084202987 (107h20m38.68097401s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.088186679 (107h20m38.684957707s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.260272609 (107h20m38.857043622s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.265905052 (107h20m38.862676079s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.281484647 (107h20m38.878255672s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.286010701 (107h20m38.882781724s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301451738 (107h20m38.89822276s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301503695 (107h20m38.898274738s) [seg_pos={0 0}, upload=120147]: Satisfied cohort requirements after 14 pieces Mar 10 22:20:41.301862444 (107h20m38.898633472s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NkSWfYxvndQHUnnQbHe6GPAHxgPe39gU84T5yHmrKKJSfivEd): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301873924 (107h20m38.898644949s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.301927973 (107h20m38.898699s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XfehFoLvcivSkVcGttnXVK5VCSWKCpmyPX3E6rtpVdwnUbbZ4): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301939854 (107h20m38.898710878s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302167039 (107h20m38.898938066s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127sHSxqrJhmDpMx2FWkzf5fXVSf7uYw2h9PWBMQCVvp2PWmqaJ): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302186195 (107h20m38.898957222s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302439932 (107h20m38.899210966s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oXSqFgqNNvDUaEwcHoEVUuaLukC2zaEfosyoeVLbXN81B3vVY): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302452647 (107h20m38.899223681s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302483492 (107h20m38.899254504s) [seg_pos={0 0}, upload=120147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:20:41.302796425 (107h20m38.899567451s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:20:41.302805132 (107h20m38.899576155s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:21:56.042886295 (107h21m53.639657321s) [upload=120147]: Done uploading segments... Mar 10 22:21:56.043103265 (107h21m53.639874297s) [upload=120147]: Upload finished. err: metaclient: context deadline exceeded Duration: 33.349563123s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...................................X.XXXXXXXXXXXXXXXXXXX.XXX] duration: 5.000635418s trace: https://jaeger.dev.storj.io/trace/7eb5de65f46fcf30 FAIL (2026-03-10 22:22:24 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000663515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.756518646s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:22:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 8.75654897s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X..X...X..XX..X] duration: 1m42.833966819s trace: https://jaeger.dev.storj.io/trace/38eb42f2a03eeafc FAIL (2026-03-10 22:21:56 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:20:13.254704710 (107h20m10.851475735s) [upload=120147]: Starting upload Mar 10 22:20:13.254766828 (107h20m10.851537845s) [upload=120147]: Done starting upload Mar 10 22:20:13.255061844 (107h20m10.851832858s) [upload=120147]: Uploading segments... Mar 10 22:20:13.255106734 (107h20m10.851877763s) [upload=120147]: Got next segment. Inline: false Mar 10 22:20:13.255154691 (107h20m10.851925714s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:20:13.255159941 (107h20m10.851930964s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:20:40.708040547 (107h20m38.304811584s) [seg_pos={0 0}, upload=120147]: Begin upload segment... Mar 10 22:20:40.708122042 (107h20m38.304893076s) [seg_pos={0 0}, upload=120147]: Done begin upload segment. Mar 10 22:20:40.708182747 (107h20m38.30495378s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708256927 (107h20m38.305027962s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.708754396 (107h20m38.30552542s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709090094 (107h20m38.305861124s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709138942 (107h20m38.305909956s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709566464 (107h20m38.306337487s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709645511 (107h20m38.306416537s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.709985832 (107h20m38.306756855s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710145510 (107h20m38.306916538s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710429544 (107h20m38.307200567s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710658858 (107h20m38.307429882s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.710837451 (107h20m38.307608473s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711140907 (107h20m38.307911939s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711249795 (107h20m38.308020814s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711720080 (107h20m38.308491106s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.711814637 (107h20m38.308585668s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712262347 (107h20m38.309033373s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.712843141 (107h20m38.309614164s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Uploading piece... Mar 10 22:20:40.778621168 (107h20m38.375392183s) [upload=120147]: Next returned nil segment Mar 10 22:20:40.778635105 (107h20m38.375406128s) [upload=120147]: Waiting for error group managing segments... Mar 10 22:20:40.783760302 (107h20m38.380531316s) [noise=true , address=79.127.219.38:4838 , piece_id=7K6GHPAO3YDL4H2MEDRMTJSQP5ATBG4L7QJDNB5G2DLUAQS6KN3A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.796179808 (107h20m38.392950836s) [noise=true , address=77.90.31.17:4801 , piece_id=62VS6I7EHDUDRUEP4K4CV6SARNKDQAAYW3PU2DOZ2EO4KFDI3UOA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800209480 (107h20m38.396980505s) [noise=true , address=77.90.31.18:4812 , piece_id=EUIQN35Z26XABNJMGP5TJL4AK2KXXKQXPMWMHTUM5NELSUT7AITA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.800815184 (107h20m38.397586196s) [noise=true , address=45.91.116.14:4207 , piece_id=AMUAUU2HANVXWVLVQLI646I57IMDQRK624GBYQSYIXBMFNNI7IJQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.807172872 (107h20m38.403943895s) [noise=true , address=145.79.227.41:4116 , piece_id=MWHO66V5HPOZK76IGYAESNAFF4B4XNDZ3V2U5G7VJV7OSCDKLFVQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:40.813580482 (107h20m38.410351505s) [noise=true , address=79.127.219.46:4897 , piece_id=DT5VFR6WN3PAHOOMD2Z6W5UNCBRIQT6Y4LI5UTA7Q7UJBUEEM4SQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.009953050 (107h20m38.606724095s) [noise=true , address=145.79.227.18:4033 , piece_id=CCS2STFBQ6CLTI32CJOH6Y52GAW5A67KLPYIF5XK5SX72IZP5RGA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.084202987 (107h20m38.68097401s) [noise=true , address=167.17.72.70:4780 , piece_id=BROYBOTVQ5QKCBHEPWOY72GVFUJKQPF3EMBYWXYJLJUTMOZD5P7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.088186679 (107h20m38.684957707s) [noise=true , address=167.17.72.69:4723 , piece_id=A3IFEEBXW453LAAKGMF5GW5F7WKUZCDOM5SFBNCHYJM5DWXRVVSQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.260272609 (107h20m38.857043622s) [noise=true , address=79.127.205.237:4220 , piece_id=RDNVD2ER2CMK4KSVOOW6EFAW67I6UITRIGV33B66NM25VBXJIGPQ, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.265905052 (107h20m38.862676079s) [noise=true , address=79.127.205.233:4176 , piece_id=HJZZSO2A5QCUVT745ZMKEW4IWKT52EIZNWHW3YQ72MLMJB2S26BA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.281484647 (107h20m38.878255672s) [noise=true , address=145.79.227.22:4540 , piece_id=HMAMBUOPFBMNVHF27YERNWNLLIZBGC7C6DQIKHXSANIZIDOAO44A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.286010701 (107h20m38.882781724s) [noise=true , address=193.149.164.71:4285 , piece_id=5YS4E2UHXNOGURPS52U3XKROL5EVEGDQ5RBKBN4IMRIEYUO5XWDA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301451738 (107h20m38.89822276s) [noise=true , address=193.149.164.66:4964 , piece_id=53XSLALMTLJIBYJADHRDL6EDLRICFLCTN7VG2AUTVW3GVXCXYZBA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: <nil> Mar 10 22:20:41.301503695 (107h20m38.898274738s) [seg_pos={0 0}, upload=120147]: Satisfied cohort requirements after 14 pieces Mar 10 22:20:41.301862444 (107h20m38.898633472s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NkSWfYxvndQHUnnQbHe6GPAHxgPe39gU84T5yHmrKKJSfivEd): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301873924 (107h20m38.898644949s) [noise=true , address=193.149.164.70:4114 , piece_id=CLR4LF7TLDWRJOXENKBJT3BJEISFF4VJVZ2LXEIKEQT2KJ2LKA2A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.301927973 (107h20m38.898699s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XfehFoLvcivSkVcGttnXVK5VCSWKCpmyPX3E6rtpVdwnUbbZ4): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.301939854 (107h20m38.898710878s) [noise=true , address=193.149.164.68:4041 , piece_id=EJ5FNR5EDHOZHFETEP7SEDEQ5O6W5OFXBL3YFFDPM5GRCYWPHLHA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302167039 (107h20m38.898938066s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127sHSxqrJhmDpMx2FWkzf5fXVSf7uYw2h9PWBMQCVvp2PWmqaJ): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302186195 (107h20m38.898957222s) [noise=true , address=193.149.164.67:4028 , piece_id=HACDXVCUMD5JNUCSRUB2527MIEI7CBHA7CLKRPMEWKZAXNDAWP7A, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302439932 (107h20m38.899210966s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oXSqFgqNNvDUaEwcHoEVUuaLukC2zaEfosyoeVLbXN81B3vVY): protocol: expected piece hash; context canceled; EOF Mar 10 22:20:41.302452647 (107h20m38.899223681s) [noise=true , address=64.49.13.139:4476 , piece_id=6F7B3UEOPD7C6PS5JY3EZMG4FM7U6NBJURMDQLJPRMG45FOBRSSA, seg_pos={0 0}, upload=120147]: Long tail cancellation due to optimal threshold Mar 10 22:20:41.302483492 (107h20m38.899254504s) [seg_pos={0 0}, upload=120147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:20:41.302796425 (107h20m38.899567451s) [upload=120147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:20:41.302805132 (107h20m38.899576155s) [upload=120147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:21:56.042886295 (107h21m53.639657321s) [upload=120147]: Done uploading segments... Mar 10 22:21:56.043103265 (107h21m53.639874297s) [upload=120147]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m42.834056899s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 48.861954725s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:21:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 48.861984943s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....................................X.XXXXXXXXXXXXXXXXXXX.XX] duration: 5.001856396s trace: https://jaeger.dev.storj.io/trace/1c2d7ee6c38048a2 FAIL (2026-03-10 22:21:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001900397s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.....................................X.XXXXXXXXXXXXXXXXXXX.X] duration: 5.000813528s trace: https://jaeger.dev.storj.io/trace/9d50b0b059f3ca FAIL (2026-03-10 22:20:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000838613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [......................................X.XXXXXXXXXXXXXXXXXXX.] duration: 5.001814923s trace: https://jaeger.dev.storj.io/trace/537b719fa608080e FAIL (2026-03-10 22:19:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001847217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X..X...X..XX..] duration: 5.216946642s trace: https://jaeger.dev.storj.io/trace/2b0370c8e16830a8 FAIL (2026-03-10 22:19:43 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:18:07.516096028 (107h18m5.112867059s) [upload=120107]: Starting upload Mar 10 22:18:07.516176377 (107h18m5.112947412s) [upload=120107]: Done starting upload Mar 10 22:18:07.516498545 (107h18m5.113269558s) [upload=120107]: Uploading segments... Mar 10 22:18:07.516557037 (107h18m5.113328066s) [upload=120107]: Got next segment. Inline: false Mar 10 22:18:07.516592873 (107h18m5.113363896s) [upload=120107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:18:07.516598619 (107h18m5.113369642s) [upload=120107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:19:07.464644724 (107h19m5.061415759s) [upload=120107]: Next segment err: context deadline exceeded Mar 10 22:19:07.464952789 (107h19m5.061723806s) [upload=120107]: Done uploading segments... Mar 10 22:19:07.465014861 (107h19m5.06178588s) [upload=120107]: Upload finished. err: context deadline exceeded Duration: 5.216978145s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 242.618355ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 242.730758ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 36.395012968s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:19:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 36.395072254s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X..X...X..XX.] duration: 16.881651928s trace: https://jaeger.dev.storj.io/trace/2b0370c8e16830a8 FAIL (2026-03-10 22:19:33 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:18:07.516096028 (107h18m5.112867059s) [upload=120107]: Starting upload Mar 10 22:18:07.516176377 (107h18m5.112947412s) [upload=120107]: Done starting upload Mar 10 22:18:07.516498545 (107h18m5.113269558s) [upload=120107]: Uploading segments... Mar 10 22:18:07.516557037 (107h18m5.113328066s) [upload=120107]: Got next segment. Inline: false Mar 10 22:18:07.516592873 (107h18m5.113363896s) [upload=120107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:18:07.516598619 (107h18m5.113369642s) [upload=120107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:19:07.464644724 (107h19m5.061415759s) [upload=120107]: Next segment err: context deadline exceeded Mar 10 22:19:07.464952789 (107h19m5.061723806s) [upload=120107]: Done uploading segments... Mar 10 22:19:07.465014861 (107h19m5.06178588s) [upload=120107]: Upload finished. err: context deadline exceeded Duration: 16.881681084s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X..X...X..XX] duration: 1m0.001810734s trace: https://jaeger.dev.storj.io/trace/2b0370c8e16830a8 FAIL (2026-03-10 22:19:07 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:18:07.516096028 (107h18m5.112867059s) [upload=120107]: Starting upload Mar 10 22:18:07.516176377 (107h18m5.112947412s) [upload=120107]: Done starting upload Mar 10 22:18:07.516498545 (107h18m5.113269558s) [upload=120107]: Uploading segments... Mar 10 22:18:07.516557037 (107h18m5.113328066s) [upload=120107]: Got next segment. Inline: false Mar 10 22:18:07.516592873 (107h18m5.113363896s) [upload=120107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:18:07.516598619 (107h18m5.113369642s) [upload=120107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:19:07.464644724 (107h19m5.061415759s) [upload=120107]: Next segment err: context deadline exceeded Mar 10 22:19:07.464952789 (107h19m5.061723806s) [upload=120107]: Done uploading segments... Mar 10 22:19:07.465014861 (107h19m5.06178588s) [upload=120107]: Upload finished. err: context deadline exceeded Duration: 1m0.001824076s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......................................X.XXXXXXXXXXXXXXXXXXX] duration: 5.000619326s trace: https://jaeger.dev.storj.io/trace/537b719fa608080e FAIL (2026-03-10 22:18:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000663806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 27.069046752s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:18:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 27.069103629s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X..X...X..X] duration: 1m0.003333136s trace: https://jaeger.dev.storj.io/trace/7bbf18cfa956d8fd FAIL (2026-03-10 22:18:07 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:17:07.874727329 (107h17m5.471498352s) [upload=120090]: Starting upload Mar 10 22:17:07.874801742 (107h17m5.471572769s) [upload=120090]: Done starting upload Mar 10 22:17:07.874999821 (107h17m5.471770855s) [upload=120090]: Uploading segments... Mar 10 22:17:07.875317864 (107h17m5.472088877s) [upload=120090]: Got next segment. Inline: false Mar 10 22:17:07.875373765 (107h17m5.472144789s) [upload=120090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:17:07.875380906 (107h17m5.472151937s) [upload=120090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:17:18.560973902 (107h17m16.157744935s) [seg_pos={0 0}, upload=120090]: Begin upload segment... Mar 10 22:17:18.561063706 (107h17m16.157849763s) [seg_pos={0 0}, upload=120090]: Done begin upload segment. Mar 10 22:17:18.561120080 (107h17m16.157891111s) [noise=true , address=45.91.116.5:4304 , piece_id=AFPWR56GDSF5G4OUYY4FFKF7HXGUFIG2S2YHPFFM6K2D2DSPCDAQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562445763 (107h17m16.159216792s) [noise=true , address=45.91.116.8:4061 , piece_id=JRCYMEVEGW5MAGMJSEC2WLZOAFOUMDYFE2MC3YLASVYXP3OV3IUQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562776154 (107h17m16.159547188s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562924616 (107h17m16.159695643s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563336583 (107h17m16.160107606s) [noise=true , address=79.127.219.38:4852 , piece_id=QJWHCVCNUPCXVYQCBGGXLWF5SMC5OR642L2YY44YFREVXBPSBLQQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563383451 (107h17m16.160154492s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563818118 (107h17m16.160589147s) [noise=true , address=45.91.116.12:4243 , piece_id=7BKWJ5FD2BHVSUH2TTIX7BV6AGPRR2Q6FQAGMSYC2T6GWLU7NPIQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564051051 (107h17m16.160822074s) [noise=true , address=145.79.227.20:4452 , piece_id=TWSG5SXDY5CUIKNFT5SKNRV7L573JKX5UD62VVJP2LDPJIH2S6MA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564283154 (107h17m16.161054187s) [noise=true , address=45.91.116.9:4270 , piece_id=YV4BFTG3ZGCMDX2XESK6ONTD3S6I45FFYUBGOB3VECRNOACAVDSQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564468819 (107h17m16.161239848s) [noise=true , address=77.90.31.11:4561 , piece_id=RFDBRIIB5HAGQOMH6VEBUXHGI6J2BVYCLELSZ35I45BDFJHNOECA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564713426 (107h17m16.161484459s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564984485 (107h17m16.161755511s) [noise=true , address=193.149.164.66:4993 , piece_id=SYRQONNIKC5VVNYWEILFC2HHI7EOP5WMQE5IXWY7EDCZLZ4L5GEQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.565410079 (107h17m16.162181104s) [noise=true , address=45.91.116.11:4125 , piece_id=G72M4EKD6XVU6TK7CHTO2PKL3TYPGXO5QCD7547D4UHOMWVR5UWA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.565843579 (107h17m16.162614602s) [noise=true , address=79.127.205.234:4206 , piece_id=DCHJK6G5QEWAWYWD3L4KGNGGN3JRJQ34KFB4JBDD3WWEMZBPVOCA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.566231863 (107h17m16.16300289s) [noise=true , address=77.90.31.16:4750 , piece_id=VK6VSZXQKYDXNQJHJ6FVJWFRGDRE324YKICJ5WUKKUUZ5V7RTGDA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.566784242 (107h17m16.163555268s) [noise=true , address=79.127.219.46:4917 , piece_id=H26ZOLMHMAUFSZUZJ4R6WD5JMURKDLPELCKZHM2S4U3CRVD7HLWA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.567341267 (107h17m16.164112293s) [noise=true , address=79.127.205.235:4269 , piece_id=BAYJ5V5Z45YZXPUAJFXNWDSLLSN4IOBZETHJ5XO335TVLBDXXEGA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.567864641 (107h17m16.16463567s) [noise=true , address=45.91.116.6:4979 , piece_id=PGWPLGE3FGWS54UTMAZIUZJYZY7V5XET75ENFGIPYIQ5DS5Q3D3Q, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.617628742 (107h17m16.214399757s) [upload=120090]: Next returned nil segment Mar 10 22:17:18.617642193 (107h17m16.214413216s) [upload=120090]: Waiting for error group managing segments... Mar 10 22:17:18.623910024 (107h17m16.220681051s) [noise=true , address=79.127.219.46:4917 , piece_id=H26ZOLMHMAUFSZUZJ4R6WD5JMURKDLPELCKZHM2S4U3CRVD7HLWA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.630973572 (107h17m16.2277446s) [noise=true , address=45.91.116.9:4270 , piece_id=YV4BFTG3ZGCMDX2XESK6ONTD3S6I45FFYUBGOB3VECRNOACAVDSQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.633165237 (107h17m16.22993625s) [noise=true , address=79.127.219.38:4852 , piece_id=QJWHCVCNUPCXVYQCBGGXLWF5SMC5OR642L2YY44YFREVXBPSBLQQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.636419787 (107h17m16.233190811s) [noise=true , address=45.91.116.5:4304 , piece_id=AFPWR56GDSF5G4OUYY4FFKF7HXGUFIG2S2YHPFFM6K2D2DSPCDAQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.638539237 (107h17m16.235310262s) [noise=true , address=45.91.116.12:4243 , piece_id=7BKWJ5FD2BHVSUH2TTIX7BV6AGPRR2Q6FQAGMSYC2T6GWLU7NPIQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.638558906 (107h17m16.23532992s) [noise=true , address=45.91.116.11:4125 , piece_id=G72M4EKD6XVU6TK7CHTO2PKL3TYPGXO5QCD7547D4UHOMWVR5UWA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.641017824 (107h17m16.237788847s) [noise=true , address=77.90.31.16:4750 , piece_id=VK6VSZXQKYDXNQJHJ6FVJWFRGDRE324YKICJ5WUKKUUZ5V7RTGDA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.664222988 (107h17m16.260994022s) [noise=true , address=145.79.227.20:4452 , piece_id=TWSG5SXDY5CUIKNFT5SKNRV7L573JKX5UD62VVJP2LDPJIH2S6MA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.669964794 (107h17m16.266735819s) [noise=true , address=77.90.31.11:4561 , piece_id=RFDBRIIB5HAGQOMH6VEBUXHGI6J2BVYCLELSZ35I45BDFJHNOECA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.693178854 (107h17m16.289949867s) [noise=true , address=45.91.116.6:4979 , piece_id=PGWPLGE3FGWS54UTMAZIUZJYZY7V5XET75ENFGIPYIQ5DS5Q3D3Q, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.731967136 (107h17m16.328738163s) [noise=true , address=45.91.116.8:4061 , piece_id=JRCYMEVEGW5MAGMJSEC2WLZOAFOUMDYFE2MC3YLASVYXP3OV3IUQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.114315886 (107h17m16.711086912s) [noise=true , address=79.127.205.234:4206 , piece_id=DCHJK6G5QEWAWYWD3L4KGNGGN3JRJQ34KFB4JBDD3WWEMZBPVOCA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.138133899 (107h17m16.734904933s) [noise=true , address=79.127.205.235:4269 , piece_id=BAYJ5V5Z45YZXPUAJFXNWDSLLSN4IOBZETHJ5XO335TVLBDXXEGA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.163335855 (107h17m16.760106878s) [noise=true , address=193.149.164.66:4993 , piece_id=SYRQONNIKC5VVNYWEILFC2HHI7EOP5WMQE5IXWY7EDCZLZ4L5GEQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.163365986 (107h17m16.76013701s) [seg_pos={0 0}, upload=120090]: Satisfied cohort requirements after 14 pieces Mar 10 22:17:19.163926502 (107h17m16.760697534s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qN4pwQ12oQxGpHgDuoFuutMUQMuLnic9syqZFsE3SKr6ZoEfT): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.163942713 (107h17m16.760713747s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164130524 (107h17m16.760901558s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ni8ECkeruDxfbM5xXepAyMEdCNjDSErZ6NP9jfGbvuy2GAdvh): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164143248 (107h17m16.760914293s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164339871 (107h17m16.761110903s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BhogobSTS5pCJxh7NGgnsU5qn1kwDDv25XxQiqqPXGSU9QL9H): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164350813 (107h17m16.761121836s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164365448 (107h17m16.761136482s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V7pTxGX19UciYsBGZxkLHqcZoG4cJdY29LtZzzA6NcF5dsiHF): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164378176 (107h17m16.761149209s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164442519 (107h17m16.761213544s) [seg_pos={0 0}, upload=120090]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:17:19.164545630 (107h17m16.761316666s) [upload=120090]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:17:19.164553464 (107h17m16.761324497s) [upload=120090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:17:38.449726305 (107h17m36.046497319s) [upload=120090]: Done uploading segments... Mar 10 22:17:38.449780753 (107h17m36.046551779s) [upload=120090]: Upload finished. err: <nil> Duration: 1m0.003360115s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [........................................X.XXXXXXXXXXXXXXXXXX] duration: 5.001161235s trace: https://jaeger.dev.storj.io/trace/382b83f429308333 FAIL (2026-03-10 22:18:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001182942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 37.981666705s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:17:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 37.981694277s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.........................................X.XXXXXXXXXXXXXXXXX] duration: 5.00126623s trace: https://jaeger.dev.storj.io/trace/1653469467a51e86 FAIL (2026-03-10 22:17:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00133922s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)