Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X........X........................................] duration:
589.597805ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:10:54 UTC)
orbiter warning: unexpected EOF
Duration: 589.622495ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [X..............X............................................] duration:
637.96579ms trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:10:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 637.99444ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.080354874s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:10:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.080423153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X........X.......................................] duration:
566.395463ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:09:48 UTC)
orbiter warning: unexpected EOF
Duration: 566.424609ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X..............X...........................................] duration:
1.123720682s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:09:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.123758731s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.030875566s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:09:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.030908797s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X........X......................................] duration:
634.342427ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:09:08 UTC)
orbiter warning: unexpected EOF
Duration: 634.372351ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.273441362s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:08:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.273477982s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X..............X..........................................] duration:
849.677779ms trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:08:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 849.704019ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X........X.....................................] duration:
732.412366ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:07:57 UTC)
orbiter warning: unexpected EOF
Duration: 732.49891ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
566.632423ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:07:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 566.667573ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
896.708996ms trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:07:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 896.734069ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............X........X....................................] duration:
953.09495ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:07:33 UTC)
orbiter warning: unexpected EOF
Duration: 953.119755ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..............X.........................................] duration:
1.065244268s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:07:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.065271877s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.238521803s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:06:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.23855313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [X...X..............X........................................] duration:
1.057218734s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:06:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.057252017s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [XX..........................................................] duration:
796.336431ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:06:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 796.365563ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
1.106750873s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:06:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.106791674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............X........X...................................] duration:
689.554624ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:06:25 UTC)
orbiter warning: unexpected EOF
Duration: 689.61502ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X...X..............X.......................................] duration:
629.890953ms trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:06:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 629.918457ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.XX.........................................................] duration:
544.433596ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:06:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.462896ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..XX........................................................] duration:
544.314657ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:05:43 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.388782ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X...X..............X......................................] duration:
1.164960367s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:05:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.165001709s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................X........X..................................] duration:
738.12804ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:05:27 UTC)
orbiter warning: unexpected EOF
Duration: 738.202916ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...XX.......................................................] duration:
552.382313ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:05:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.413756ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
1.259828648s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:05:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.259911819s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.187434458s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:05:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.187465947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
918.513916ms trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:04:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 918.558547ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................X........X.................................] duration:
723.556806ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:04:46 UTC)
orbiter warning: unexpected EOF
Duration: 723.580293ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X...XX......................................................] duration:
724.802714ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:04:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 724.838145ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..X...X..............X.....................................] duration:
1.215125057s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:04:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.215152183s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [XX...XX.....................................................] duration:
606.687498ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:04:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 606.710712ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
1.272743062s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:03:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.27277283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................X........X................................] duration:
959.250079ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:03:51 UTC)
orbiter warning: unexpected EOF
Duration: 959.275561ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX..X...X..............X....................................] duration:
909.625807ms trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:03:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 909.651984ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [XXX...XX....................................................] duration:
604.494579ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:03:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 604.517324ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
1.332021977s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:03:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.332050952s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
1.200353897s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:02:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.200442876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................X........X...............................] duration:
566.763295ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:02:41 UTC)
orbiter warning: unexpected EOF
Duration: 566.790001ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.XXX...XX...................................................] duration:
484.604495ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:02:30 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.637665ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX..X...X..............X...................................] duration:
1.162873577s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:02:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.162939509s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
1.19843697s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:01:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.198460925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [X.XXX...XX..................................................] duration:
528.391559ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:01:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.413278ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX..X...X..............X..................................] duration:
1.084714708s trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:01:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 1.084745386s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................X........X..............................] duration:
695.37026ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:01:26 UTC)
orbiter warning: unexpected EOF
Duration: 695.397952ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
1.073691528s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:01:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.073719987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [XX.XXX...XX.................................................] duration:
520.53825ms trace:
https://jaeger.dev.storj.io/trace/6a3ff061c0ba5435
FAIL (2026-03-10 23:01:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 520.565619ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [X..XX..X...X..............X.................................] duration:
876.847713ms trace:
https://jaeger.dev.storj.io/trace/30625685be2c4b28
FAIL (2026-03-10 23:01:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Mar 10 22:36:48.773908080 (107h36m46.370679103s) [upload=120456]: Starting upload
Mar 10 22:36:48.773962988 (107h36m46.370734014s) [upload=120456]: Done starting upload
Mar 10 22:36:48.774066777 (107h36m46.37083779s) [upload=120456]: Uploading segments...
Mar 10 22:36:48.774366193 (107h36m46.371137207s) [upload=120456]: Got next segment. Inline: false
Mar 10 22:36:48.774431356 (107h36m46.371202369s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 10 22:36:48.774438092 (107h36m46.371209115s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 10 22:36:48.805695910 (107h36m46.402466924s) [seg_pos={0 0}, upload=120456]: Begin upload segment...
Mar 10 22:36:48.805794912 (107h36m46.402565935s) [seg_pos={0 0}, upload=120456]: Done begin upload segment.
Mar 10 22:36:48.805809136 (107h36m46.402580169s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.805841205 (107h36m46.402612238s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806573553 (107h36m46.403344585s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.806937623 (107h36m46.403708649s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808139115 (107h36m46.404910141s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.808720779 (107h36m46.405491803s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809205963 (107h36m46.405976987s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.809735148 (107h36m46.406506172s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.811470205 (107h36m46.408241235s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812209754 (107h36m46.408980784s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.812910217 (107h36m46.409681254s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.813856431 (107h36m46.410627464s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.814512122 (107h36m46.411283156s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815156222 (107h36m46.411927252s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.815865007 (107h36m46.412636037s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816422088 (107h36m46.413193099s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.816970708 (107h36m46.413741742s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.817200854 (107h36m46.413971888s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Uploading piece...
Mar 10 22:36:48.889689994 (107h36m46.486461024s) [upload=120456]: Next returned nil segment
Mar 10 22:36:48.889713533 (107h36m46.486484563s) [upload=120456]: Waiting for error group managing segments...
Mar 10 22:36:48.905028030 (107h36m46.501799058s) [noise=true , address=79.127.219.46:4924 , piece_id=GSIWRYMMC5P3VW6KHUA5IQXTFN6Q7V6D5ZJ7YHBXJVDDPZ27BPLA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.925066540 (107h36m46.521837553s) [noise=true , address=145.79.227.39:4030 , piece_id=5HVILI5WUX3WUYWNX6WCAIPOURIBPWAGN3K6REOHLTR7ZTV6AE2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:48.988560128 (107h36m46.585331155s) [noise=true , address=5.231.73.3:4018 , piece_id=BSDLLJ2W6XOQ4DDNFOEPBZYDZ6A4VQTQ2U7RJU5AXWTZFXK3GKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.357089448 (107h36m46.953860462s) [noise=true , address=77.90.31.10:4540 , piece_id=U5O5WQZ3ZMWDUPMW373OGSAAGWRTSSYZTOUHBVHEZ3YI5PCOV7HA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.465656199 (107h36m47.062427243s) [noise=true , address=77.90.31.12:4596 , piece_id=LRKCOB7LJVGTYX3AYXFCXOXWHKQHPAFGJOOIQYKFJVZKRXIR2AOA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.474429145 (107h36m47.071200159s) [noise=true , address=64.49.13.155:4916 , piece_id=TRWEMLREKCV5TGAORLNU77Y7MGKTM4HWRJ5VKF67XMSM6PMC22DQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.610670188 (107h36m47.207441215s) [noise=true , address=45.91.116.11:4123 , piece_id=AQ64FT2MC4XF54T2RMDOCZFLVM5YTFLUDIEIFRCVR3V3K4GHUQLQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.658014416 (107h36m47.25478543s) [noise=true , address=167.17.72.70:4775 , piece_id=XF6X7HPJ3WO2HFEYAER2XK66X4F5BOIFQGO4L7BCZEYUQFLJPFSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.668839985 (107h36m47.265610995s) [noise=true , address=77.90.31.19:4846 , piece_id=LXFGEIHQFESEIHRPS7AETUBWOMFNLN326Z6U3PM6GJTSQRMAA7QQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.686150399 (107h36m47.282921442s) [noise=true , address=77.90.31.18:4828 , piece_id=TE3VPMJ7FKCBSXJ34BSX62VBXTCFICXRGYXPCIKAHNRXBPJJB5RA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.779956250 (107h36m47.376727273s) [noise=true , address=64.49.13.138:4488 , piece_id=EWOTTTXMTMOSGMH4KUW3TVFP2VZWDN5AAG3EXOCL4KRQ7G3ZOZFA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:49.993624811 (107h36m47.590395839s) [noise=true , address=77.90.31.15:4729 , piece_id=XUX234NGVTHJYDXZBYTZ2EADW7XOWTKFKHLAMUQROEO6CZK2U5HQ, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.204438859 (107h36m47.801209869s) [noise=true , address=45.91.116.8:4065 , piece_id=TCQMR7FPWPU6MWKHQKADFN72VYEEKNY3UBJFEKOOWS56GF2AZJ5Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669817054 (107h36m48.266588078s) [noise=true , address=45.91.116.13:4179 , piece_id=XIQVPTK255YDASECJZZ6BVDELIAYNHTO4SNGTORZFLRPY73KSK2Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: <nil>
Mar 10 22:36:50.669878422 (107h36m48.266649445s) [seg_pos={0 0}, upload=120456]: Satisfied cohort requirements after 14 pieces
Mar 10 22:36:50.670293943 (107h36m48.267064969s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zzK691zWMo8NYsaehHwjwe2SN2ivcrf5bVA6vjMrW3Y9UtWBU): protocol: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48944->193.149.164.70:4111: use of closed network connection
Mar 10 22:36:50.670303843 (107h36m48.267074868s) [noise=true , address=193.149.164.70:4111 , piece_id=SVV7PNRB7YSDDTMA5T7VIIN5PKIUG3PAAX67QSVXQUBTDXZLMKSA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.670632276 (107h36m48.267403312s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi): protocol: context canceled; context canceled
Mar 10 22:36:50.670642107 (107h36m48.267413135s) [noise=true , address=45.91.116.14:4198 , piece_id=ZLGAKUTYT37RA6FT5RNN3JVWAWPVNKBM4EC4BPDFX6JICXSOIO4A, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671322841 (107h36m48.268093871s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i3PBJEuA51nnHNnYjiR7bMPBotipQaAfNrU1WD1YWHJf74LSz): protocol: read tcp 159.223.143.39:57260->193.149.164.69:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:36:50.671336064 (107h36m48.268107095s) [noise=true , address=193.149.164.69:4102 , piece_id=5D6UKJH5RQM7N5VNYCV7FKRRVZOS5QBLTX73PDHR5DRPDM2VIK4Q, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671400395 (107h36m48.268171411s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Mar 10 22:36:50.671413826 (107h36m48.268184849s) [noise=true , address=77.90.31.16:4760 , piece_id=JFYMBJUQV4ST2UUQCS2UEDNGK6AR36PC3WQ2AJBQUGZIGD6CSTNA, seg_pos={0 0}, upload=120456]: Long tail cancellation due to optimal threshold
Mar 10 22:36:50.671528955 (107h36m48.268299968s) [seg_pos={0 0}, upload=120456]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:36:50.671623368 (107h36m48.268394391s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:36:50.671629202 (107h36m48.268400224s) [upload=120456]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:36:50.725412496 (107h36m48.322183522s) [upload=120456]: Done uploading segments...
Mar 10 22:36:50.725435111 (107h36m48.322206135s) [upload=120456]: Upload finished. err: <nil>
Duration: 876.876134ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................X........X.............................] duration:
761.333371ms trace:
https://jaeger.dev.storj.io/trace/26921813beafaaf8
FAIL (2026-03-10 23:00:17 UTC)
orbiter warning: unexpected EOF
Duration: 761.359864ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
1.156018026s trace:
https://jaeger.dev.storj.io/trace/58132c67a664c204
FAIL (2026-03-10 23:00:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
CU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958261103 (107h26m38.555032128s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958589592 (107h26m38.555360619s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.958881731 (107h26m38.555652757s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959439527 (107h26m38.556210553s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959552651 (107h26m38.556323677s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.959934704 (107h26m38.556705727s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960103743 (107h26m38.556874778s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960291294 (107h26m38.557062318s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960592015 (107h26m38.55736305s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960651567 (107h26m38.55742259s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.960986965 (107h26m38.557757989s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961058558 (107h26m38.557829571s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961325774 (107h26m38.558096798s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961465028 (107h26m38.558236051s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:40.961763034 (107h26m38.55853406s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Uploading piece...
Mar 10 22:26:41.022086485 (107h26m38.618857518s) [upload=120263]: Next returned nil segment
Mar 10 22:26:41.022109635 (107h26m38.618880662s) [upload=120263]: Waiting for error group managing segments...
Mar 10 22:26:41.028507375 (107h26m38.625278398s) [noise=true , address=79.127.219.38:4837 , piece_id=NERCN4YAN4YXBBSCHNS7HTEZZM6JZHNYGWJNTGRWP2GUIO5FZJDQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.046710482 (107h26m38.643481512s) [noise=true , address=145.79.227.33:4877 , piece_id=SYI464ARWRQIPNBRBC7XYBQTNNT44Y6FETTNJOKFA6RFSA62IPHA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.050752974 (107h26m38.647523989s) [noise=true , address=145.79.227.46:4612 , piece_id=EAUHH4QKASHIF6AIFRRTIUUEMMFM5BL5XZO25BT4C5DQX6WUVHIA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.068687281 (107h26m38.665458305s) [noise=true , address=145.79.227.15:4890 , piece_id=QHBMKEKSAXF65PQL4O4ZWY2N72IS6WS3NFHEQWAXILBOQDTQ7PFA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.091072409 (107h26m38.687843433s) [noise=true , address=45.91.116.10:4112 , piece_id=G4LOAJEZCU6FNMV2UZQRI7DXTSSAUNA6S2QCQKBWTPYLODFWSLPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.188164769 (107h26m38.784935792s) [noise=true , address=77.90.31.10:4524 , piece_id=M6V7OGUCDAY6GXUL65ITXD4ISIF4TKDEJ35XCHHMPNGKUPCMY2MA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.258304300 (107h26m38.855075329s) [noise=true , address=45.91.116.11:4123 , piece_id=WIZHCPK7E4OVYVOHVEMHBGVJTXKUGPWI57E7OU7R7MHJN7XVC72A, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.272518764 (107h26m38.869289802s) [noise=true , address=45.91.116.7:4039 , piece_id=XHNUAQVZIUFO6KMHJZ7JO4UR4WTCSY33MRR5M4PY65SZ2LRTEFPQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.445904384 (107h26m39.042675408s) [noise=true , address=45.91.116.12:4247 , piece_id=WZJQX6NNYFGCDO7M7TQFYYRGRGHD2TAF7T3KLIHHOIKVA7GT45KA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.448685024 (107h26m39.045456054s) [noise=true , address=77.90.31.13:4653 , piece_id=SAKTBNNN4NRAM3MF7K5Y3XZTGDWGC7K6WZ2M4F7JIFBLLX34WKCQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.536106023 (107h26m39.132877051s) [noise=true , address=79.127.205.234:4210 , piece_id=ELCWTDS5ACMXLHBL4GXDBGESK5LXANZMAO77CMCQW2I6BCOUQCSQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.539348419 (107h26m39.136119453s) [noise=true , address=79.127.205.237:4239 , piece_id=47ELQT7O7RCL6LZ2GXY4IRFLJMUN7EHSSVNLYGFC7ZECGRHYJGKA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.557949063 (107h26m39.154720077s) [noise=true , address=79.127.205.233:4143 , piece_id=6ZQUU7B6MV4JHYXFSTP4TTXNDWES33BC2HTRJRZZXFCBBDTYVMTQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642579444 (107h26m39.239350471s) [noise=true , address=193.149.164.143:4697 , piece_id=Q64HJVI7SPSAVAE4N7ZSGJTJRMIU5U2DLDLWFCT7VWRRZGO3GSXA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: <nil>
Mar 10 22:26:41.642604575 (107h26m39.239375601s) [seg_pos={0 0}, upload=120263]: Satisfied cohort requirements after 14 pieces
Mar 10 22:26:41.643838671 (107h26m39.240609704s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9): protocol: context canceled; context canceled
Mar 10 22:26:41.643871441 (107h26m39.240642473s) [noise=true , address=167.17.72.71:4811 , piece_id=UWUHPSGO43OC26JDMMFAD4YNV6MUMDH3KB6FBKVNSEFPF5WLSCUA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644130921 (107h26m39.240901952s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8Av3D2aNHTzxRgFU3iHMiRETveHgcB6L2RQ6DDoiwih98u3f): protocol: read tcp 159.223.143.39:54054->193.149.164.69:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644138810 (107h26m39.24090984s) [noise=true , address=193.149.164.69:4100 , piece_id=7CSQMO4KF3GCWLKU5XVJP6COCBQ7F4ICQFSNW7BG6CYXNB4ZJRBA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.644762810 (107h26m39.241533845s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srStVhazXKryuAYTSnrzbYRyxR93grixD2YVVPEoQdrBEXeBU): protocol: read tcp 159.223.143.39:54218->193.149.164.71:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.644796803 (107h26m39.241567827s) [noise=true , address=193.149.164.71:4297 , piece_id=N7KBMZHKPEELP3XEDAKQAIIO7EQWOF2DC2DT257UKB7BC2OJ62PA, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645027359 (107h26m39.241798382s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: read tcp 159.223.143.39:35606->193.149.164.67:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 10 22:26:41.645035006 (107h26m39.241806032s) [noise=true , address=193.149.164.67:4007 , piece_id=C3GF47ZJ6ETLZFZD2V4KAU46AL7SEQYZM46VPKY6TNJ5SALRJFWQ, seg_pos={0 0}, upload=120263]: Long tail cancellation due to optimal threshold
Mar 10 22:26:41.645079629 (107h26m39.241850645s) [seg_pos={0 0}, upload=120263]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 10 22:26:41.645290037 (107h26m39.242061061s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 10 22:26:41.645296447 (107h26m39.24206747s) [upload=120263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 10 22:26:41.693386878 (107h26m39.29015791s) [upload=120263]: Done uploading segments...
Mar 10 22:26:41.693571126 (107h26m39.290342149s) [upload=120263]: Upload finished. err: <nil>
Duration: 1.156051282s
core
uplink
us-select-1
us1-placement-20
|