Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 320304 reports (Page 388 of 6407)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 242.618355ms trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 242.730758ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 36.395012968s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:19:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 36.395072254s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X..X...X..XX.] duration: 16.881651928s trace: https://jaeger.dev.storj.io/trace/2b0370c8e16830a8 FAIL (2026-03-10 22:19:33 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:18:07.516096028 (107h18m5.112867059s) [upload=120107]: Starting upload Mar 10 22:18:07.516176377 (107h18m5.112947412s) [upload=120107]: Done starting upload Mar 10 22:18:07.516498545 (107h18m5.113269558s) [upload=120107]: Uploading segments... Mar 10 22:18:07.516557037 (107h18m5.113328066s) [upload=120107]: Got next segment. Inline: false Mar 10 22:18:07.516592873 (107h18m5.113363896s) [upload=120107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:18:07.516598619 (107h18m5.113369642s) [upload=120107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:19:07.464644724 (107h19m5.061415759s) [upload=120107]: Next segment err: context deadline exceeded Mar 10 22:19:07.464952789 (107h19m5.061723806s) [upload=120107]: Done uploading segments... Mar 10 22:19:07.465014861 (107h19m5.06178588s) [upload=120107]: Upload finished. err: context deadline exceeded Duration: 16.881681084s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X..X...X..XX] duration: 1m0.001810734s trace: https://jaeger.dev.storj.io/trace/2b0370c8e16830a8 FAIL (2026-03-10 22:19:07 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:18:07.516096028 (107h18m5.112867059s) [upload=120107]: Starting upload Mar 10 22:18:07.516176377 (107h18m5.112947412s) [upload=120107]: Done starting upload Mar 10 22:18:07.516498545 (107h18m5.113269558s) [upload=120107]: Uploading segments... Mar 10 22:18:07.516557037 (107h18m5.113328066s) [upload=120107]: Got next segment. Inline: false Mar 10 22:18:07.516592873 (107h18m5.113363896s) [upload=120107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:18:07.516598619 (107h18m5.113369642s) [upload=120107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:19:07.464644724 (107h19m5.061415759s) [upload=120107]: Next segment err: context deadline exceeded Mar 10 22:19:07.464952789 (107h19m5.061723806s) [upload=120107]: Done uploading segments... Mar 10 22:19:07.465014861 (107h19m5.06178588s) [upload=120107]: Upload finished. err: context deadline exceeded Duration: 1m0.001824076s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......................................X.XXXXXXXXXXXXXXXXXXX] duration: 5.000619326s trace: https://jaeger.dev.storj.io/trace/537b719fa608080e FAIL (2026-03-10 22:18:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000663806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 27.069046752s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:18:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 27.069103629s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X..X...X..X] duration: 1m0.003333136s trace: https://jaeger.dev.storj.io/trace/7bbf18cfa956d8fd FAIL (2026-03-10 22:18:07 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:17:07.874727329 (107h17m5.471498352s) [upload=120090]: Starting upload Mar 10 22:17:07.874801742 (107h17m5.471572769s) [upload=120090]: Done starting upload Mar 10 22:17:07.874999821 (107h17m5.471770855s) [upload=120090]: Uploading segments... Mar 10 22:17:07.875317864 (107h17m5.472088877s) [upload=120090]: Got next segment. Inline: false Mar 10 22:17:07.875373765 (107h17m5.472144789s) [upload=120090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:17:07.875380906 (107h17m5.472151937s) [upload=120090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:17:18.560973902 (107h17m16.157744935s) [seg_pos={0 0}, upload=120090]: Begin upload segment... Mar 10 22:17:18.561063706 (107h17m16.157849763s) [seg_pos={0 0}, upload=120090]: Done begin upload segment. Mar 10 22:17:18.561120080 (107h17m16.157891111s) [noise=true , address=45.91.116.5:4304 , piece_id=AFPWR56GDSF5G4OUYY4FFKF7HXGUFIG2S2YHPFFM6K2D2DSPCDAQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562445763 (107h17m16.159216792s) [noise=true , address=45.91.116.8:4061 , piece_id=JRCYMEVEGW5MAGMJSEC2WLZOAFOUMDYFE2MC3YLASVYXP3OV3IUQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562776154 (107h17m16.159547188s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.562924616 (107h17m16.159695643s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563336583 (107h17m16.160107606s) [noise=true , address=79.127.219.38:4852 , piece_id=QJWHCVCNUPCXVYQCBGGXLWF5SMC5OR642L2YY44YFREVXBPSBLQQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563383451 (107h17m16.160154492s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.563818118 (107h17m16.160589147s) [noise=true , address=45.91.116.12:4243 , piece_id=7BKWJ5FD2BHVSUH2TTIX7BV6AGPRR2Q6FQAGMSYC2T6GWLU7NPIQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564051051 (107h17m16.160822074s) [noise=true , address=145.79.227.20:4452 , piece_id=TWSG5SXDY5CUIKNFT5SKNRV7L573JKX5UD62VVJP2LDPJIH2S6MA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564283154 (107h17m16.161054187s) [noise=true , address=45.91.116.9:4270 , piece_id=YV4BFTG3ZGCMDX2XESK6ONTD3S6I45FFYUBGOB3VECRNOACAVDSQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564468819 (107h17m16.161239848s) [noise=true , address=77.90.31.11:4561 , piece_id=RFDBRIIB5HAGQOMH6VEBUXHGI6J2BVYCLELSZ35I45BDFJHNOECA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564713426 (107h17m16.161484459s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.564984485 (107h17m16.161755511s) [noise=true , address=193.149.164.66:4993 , piece_id=SYRQONNIKC5VVNYWEILFC2HHI7EOP5WMQE5IXWY7EDCZLZ4L5GEQ, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.565410079 (107h17m16.162181104s) [noise=true , address=45.91.116.11:4125 , piece_id=G72M4EKD6XVU6TK7CHTO2PKL3TYPGXO5QCD7547D4UHOMWVR5UWA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.565843579 (107h17m16.162614602s) [noise=true , address=79.127.205.234:4206 , piece_id=DCHJK6G5QEWAWYWD3L4KGNGGN3JRJQ34KFB4JBDD3WWEMZBPVOCA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.566231863 (107h17m16.16300289s) [noise=true , address=77.90.31.16:4750 , piece_id=VK6VSZXQKYDXNQJHJ6FVJWFRGDRE324YKICJ5WUKKUUZ5V7RTGDA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.566784242 (107h17m16.163555268s) [noise=true , address=79.127.219.46:4917 , piece_id=H26ZOLMHMAUFSZUZJ4R6WD5JMURKDLPELCKZHM2S4U3CRVD7HLWA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.567341267 (107h17m16.164112293s) [noise=true , address=79.127.205.235:4269 , piece_id=BAYJ5V5Z45YZXPUAJFXNWDSLLSN4IOBZETHJ5XO335TVLBDXXEGA, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.567864641 (107h17m16.16463567s) [noise=true , address=45.91.116.6:4979 , piece_id=PGWPLGE3FGWS54UTMAZIUZJYZY7V5XET75ENFGIPYIQ5DS5Q3D3Q, seg_pos={0 0}, upload=120090]: Uploading piece... Mar 10 22:17:18.617628742 (107h17m16.214399757s) [upload=120090]: Next returned nil segment Mar 10 22:17:18.617642193 (107h17m16.214413216s) [upload=120090]: Waiting for error group managing segments... Mar 10 22:17:18.623910024 (107h17m16.220681051s) [noise=true , address=79.127.219.46:4917 , piece_id=H26ZOLMHMAUFSZUZJ4R6WD5JMURKDLPELCKZHM2S4U3CRVD7HLWA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.630973572 (107h17m16.2277446s) [noise=true , address=45.91.116.9:4270 , piece_id=YV4BFTG3ZGCMDX2XESK6ONTD3S6I45FFYUBGOB3VECRNOACAVDSQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.633165237 (107h17m16.22993625s) [noise=true , address=79.127.219.38:4852 , piece_id=QJWHCVCNUPCXVYQCBGGXLWF5SMC5OR642L2YY44YFREVXBPSBLQQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.636419787 (107h17m16.233190811s) [noise=true , address=45.91.116.5:4304 , piece_id=AFPWR56GDSF5G4OUYY4FFKF7HXGUFIG2S2YHPFFM6K2D2DSPCDAQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.638539237 (107h17m16.235310262s) [noise=true , address=45.91.116.12:4243 , piece_id=7BKWJ5FD2BHVSUH2TTIX7BV6AGPRR2Q6FQAGMSYC2T6GWLU7NPIQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.638558906 (107h17m16.23532992s) [noise=true , address=45.91.116.11:4125 , piece_id=G72M4EKD6XVU6TK7CHTO2PKL3TYPGXO5QCD7547D4UHOMWVR5UWA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.641017824 (107h17m16.237788847s) [noise=true , address=77.90.31.16:4750 , piece_id=VK6VSZXQKYDXNQJHJ6FVJWFRGDRE324YKICJ5WUKKUUZ5V7RTGDA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.664222988 (107h17m16.260994022s) [noise=true , address=145.79.227.20:4452 , piece_id=TWSG5SXDY5CUIKNFT5SKNRV7L573JKX5UD62VVJP2LDPJIH2S6MA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.669964794 (107h17m16.266735819s) [noise=true , address=77.90.31.11:4561 , piece_id=RFDBRIIB5HAGQOMH6VEBUXHGI6J2BVYCLELSZ35I45BDFJHNOECA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.693178854 (107h17m16.289949867s) [noise=true , address=45.91.116.6:4979 , piece_id=PGWPLGE3FGWS54UTMAZIUZJYZY7V5XET75ENFGIPYIQ5DS5Q3D3Q, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:18.731967136 (107h17m16.328738163s) [noise=true , address=45.91.116.8:4061 , piece_id=JRCYMEVEGW5MAGMJSEC2WLZOAFOUMDYFE2MC3YLASVYXP3OV3IUQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.114315886 (107h17m16.711086912s) [noise=true , address=79.127.205.234:4206 , piece_id=DCHJK6G5QEWAWYWD3L4KGNGGN3JRJQ34KFB4JBDD3WWEMZBPVOCA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.138133899 (107h17m16.734904933s) [noise=true , address=79.127.205.235:4269 , piece_id=BAYJ5V5Z45YZXPUAJFXNWDSLLSN4IOBZETHJ5XO335TVLBDXXEGA, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.163335855 (107h17m16.760106878s) [noise=true , address=193.149.164.66:4993 , piece_id=SYRQONNIKC5VVNYWEILFC2HHI7EOP5WMQE5IXWY7EDCZLZ4L5GEQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: <nil> Mar 10 22:17:19.163365986 (107h17m16.76013701s) [seg_pos={0 0}, upload=120090]: Satisfied cohort requirements after 14 pieces Mar 10 22:17:19.163926502 (107h17m16.760697534s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qN4pwQ12oQxGpHgDuoFuutMUQMuLnic9syqZFsE3SKr6ZoEfT): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.163942713 (107h17m16.760713747s) [noise=true , address=64.49.13.139:4480 , piece_id=DBFEOANMWEIB7GW3YPQCNTHDV4VPRUVJPEY3YRZ3KWV47RBKZTUQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164130524 (107h17m16.760901558s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ni8ECkeruDxfbM5xXepAyMEdCNjDSErZ6NP9jfGbvuy2GAdvh): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164143248 (107h17m16.760914293s) [noise=true , address=193.149.164.69:4084 , piece_id=6CX5XCO2NGN5F7DHV3I5RBB3AV6VWYZW6XH5C5RLEIFGLE6FKAVQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164339871 (107h17m16.761110903s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BhogobSTS5pCJxh7NGgnsU5qn1kwDDv25XxQiqqPXGSU9QL9H): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164350813 (107h17m16.761121836s) [noise=true , address=193.149.164.67:4020 , piece_id=7BSV6RRWIBULA5W26U75DTZIQMUBVQIRLDBBZEZBRM4NAKNKJYGQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164365448 (107h17m16.761136482s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V7pTxGX19UciYsBGZxkLHqcZoG4cJdY29LtZzzA6NcF5dsiHF): protocol: expected piece hash; context canceled; EOF Mar 10 22:17:19.164378176 (107h17m16.761149209s) [noise=true , address=193.149.164.71:4305 , piece_id=DKL6PY2ATXQ5SCEPNVFVE4PBFZUNQTA3DELNCZMHI4NAM4UWC2QQ, seg_pos={0 0}, upload=120090]: Long tail cancellation due to optimal threshold Mar 10 22:17:19.164442519 (107h17m16.761213544s) [seg_pos={0 0}, upload=120090]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:17:19.164545630 (107h17m16.761316666s) [upload=120090]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:17:19.164553464 (107h17m16.761324497s) [upload=120090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:17:38.449726305 (107h17m36.046497319s) [upload=120090]: Done uploading segments... Mar 10 22:17:38.449780753 (107h17m36.046551779s) [upload=120090]: Upload finished. err: <nil> Duration: 1m0.003360115s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [........................................X.XXXXXXXXXXXXXXXXXX] duration: 5.001161235s trace: https://jaeger.dev.storj.io/trace/382b83f429308333 FAIL (2026-03-10 22:18:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001182942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 37.981666705s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:17:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 37.981694277s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.........................................X.XXXXXXXXXXXXXXXXX] duration: 5.00126623s trace: https://jaeger.dev.storj.io/trace/1653469467a51e86 FAIL (2026-03-10 22:17:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00133922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..........................................X.XXXXXXXXXXXXXXXX] duration: 5.00068977s trace: https://jaeger.dev.storj.io/trace/7050390600fa8d48 FAIL (2026-03-10 22:16:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000704174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 25.28868075s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:16:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 25.288702986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X..X...X..] duration: 1.317151386s trace: https://jaeger.dev.storj.io/trace/24ae29829d7f473c FAIL (2026-03-10 22:16:15 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:14:57.782559119 (107h14m55.379330167s) [upload=120051]: Starting upload Mar 10 22:14:57.782648188 (107h14m55.379419238s) [upload=120051]: Done starting upload Mar 10 22:14:57.782756637 (107h14m55.37952766s) [upload=120051]: Uploading segments... Mar 10 22:14:57.783080239 (107h14m55.379851255s) [upload=120051]: Got next segment. Inline: false Mar 10 22:14:57.783170765 (107h14m55.379941788s) [upload=120051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:14:57.783177273 (107h14m55.379948297s) [upload=120051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:15:57.733426415 (107h15m55.330197472s) [upload=120051]: Next segment err: context deadline exceeded Mar 10 22:15:57.733699737 (107h15m55.330470761s) [upload=120051]: Done uploading segments... Mar 10 22:15:57.733756300 (107h15m55.330527325s) [upload=120051]: Upload finished. err: context deadline exceeded Duration: 1.317175354s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X..X...X.] duration: 15.307867574s trace: https://jaeger.dev.storj.io/trace/24ae29829d7f473c FAIL (2026-03-10 22:16:13 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:14:57.782559119 (107h14m55.379330167s) [upload=120051]: Starting upload Mar 10 22:14:57.782648188 (107h14m55.379419238s) [upload=120051]: Done starting upload Mar 10 22:14:57.782756637 (107h14m55.37952766s) [upload=120051]: Uploading segments... Mar 10 22:14:57.783080239 (107h14m55.379851255s) [upload=120051]: Got next segment. Inline: false Mar 10 22:14:57.783170765 (107h14m55.379941788s) [upload=120051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:14:57.783177273 (107h14m55.379948297s) [upload=120051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:15:57.733426415 (107h15m55.330197472s) [upload=120051]: Next segment err: context deadline exceeded Mar 10 22:15:57.733699737 (107h15m55.330470761s) [upload=120051]: Done uploading segments... Mar 10 22:15:57.733756300 (107h15m55.330527325s) [upload=120051]: Upload finished. err: context deadline exceeded Duration: 15.307887087s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 30.186085457s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:16:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 30.186130616s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...........................................X.XXXXXXXXXXXXXXX] duration: 5.00072537s trace: https://jaeger.dev.storj.io/trace/14e672d5539c0936 FAIL (2026-03-10 22:16:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000745935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X..X...X] duration: 1m0.001024634s trace: https://jaeger.dev.storj.io/trace/24ae29829d7f473c FAIL (2026-03-10 22:15:57 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 10 22:14:57.782559119 (107h14m55.379330167s) [upload=120051]: Starting upload Mar 10 22:14:57.782648188 (107h14m55.379419238s) [upload=120051]: Done starting upload Mar 10 22:14:57.782756637 (107h14m55.37952766s) [upload=120051]: Uploading segments... Mar 10 22:14:57.783080239 (107h14m55.379851255s) [upload=120051]: Got next segment. Inline: false Mar 10 22:14:57.783170765 (107h14m55.379941788s) [upload=120051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:14:57.783177273 (107h14m55.379948297s) [upload=120051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:15:57.733426415 (107h15m55.330197472s) [upload=120051]: Next segment err: context deadline exceeded Mar 10 22:15:57.733699737 (107h15m55.330470761s) [upload=120051]: Done uploading segments... Mar 10 22:15:57.733756300 (107h15m55.330527325s) [upload=120051]: Upload finished. err: context deadline exceeded Duration: 1m0.001039871s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 24.64495044s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:15:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 24.644979793s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration: 47.616133235s trace: https://jaeger.dev.storj.io/trace/236bcdc90d857f3c FAIL (2026-03-10 22:14:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 10 22:11:38.970020774 (107h11m36.566791803s) [upload=119992]: Starting upload Mar 10 22:11:38.970116316 (107h11m36.566887331s) [upload=119992]: Done starting upload Mar 10 22:11:38.970424312 (107h11m36.567195326s) [upload=119992]: Uploading segments... Mar 10 22:11:38.970574854 (107h11m36.567345885s) [upload=119992]: Got next segment. Inline: false Mar 10 22:11:38.970650119 (107h11m36.567421134s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:38.970657466 (107h11m36.567428491s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:11:41.578785710 (107h11m39.175556741s) [seg_pos={0 0}, upload=119992]: Begin upload segment... Mar 10 22:11:41.578861456 (107h11m39.175632482s) [seg_pos={0 0}, upload=119992]: Done begin upload segment. Mar 10 22:11:41.578910683 (107h11m39.175681716s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.578926373 (107h11m39.175697402s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579585265 (107h11m39.176356295s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579908593 (107h11m39.176679619s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580190868 (107h11m39.176961906s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580310098 (107h11m39.17708112s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580644652 (107h11m39.177415681s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580658828 (107h11m39.177429864s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581041094 (107h11m39.177812122s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581099790 (107h11m39.177870825s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581480438 (107h11m39.178251469s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581532640 (107h11m39.178303668s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581870926 (107h11m39.178641952s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581963946 (107h11m39.178734979s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582201392 (107h11m39.178972424s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582405729 (107h11m39.179176767s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582870960 (107h11m39.179641996s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.583377134 (107h11m39.180148161s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.659615112 (107h11m39.256386141s) [upload=119992]: Next returned nil segment Mar 10 22:11:41.659705421 (107h11m39.256476449s) [upload=119992]: Waiting for error group managing segments... Mar 10 22:11:41.673205837 (107h11m39.269976868s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677131898 (107h11m39.273902921s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677231485 (107h11m39.274002513s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680107367 (107h11m39.276878391s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680874797 (107h11m39.277645826s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.684631539 (107h11m39.281402562s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.685306601 (107h11m39.282077629s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.707110296 (107h11m39.303881323s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.708508816 (107h11m39.305279832s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.779617247 (107h11m39.37638827s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.970373836 (107h11m39.567144863s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.129282623 (107h11m39.726053635s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.160167419 (107h11m39.756938442s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173664351 (107h11m39.770435374s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173688329 (107h11m39.770459353s) [seg_pos={0 0}, upload=119992]: Satisfied cohort requirements after 14 pieces Mar 10 22:11:42.173944215 (107h11m39.77071525s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.173954485 (107h11m39.770725509s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174094844 (107h11m39.770865874s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sK7oayGYBqB4fhjodVcZLid7438rnseaGQqTuZ4pTU5YNSuHi): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174110699 (107h11m39.770881729s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174127203 (107h11m39.770898235s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n739DCDptMWxwBZJ9sGaZLWbpYrCEFQRk8UDZe6QqrYmhGofF): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174133958 (107h11m39.770904981s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174283426 (107h11m39.771054454s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uH4iMUgUmHa22vDB92ERVX4G7PDch6T2zLw8ekGMiUxMiB3b2): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174291249 (107h11m39.771062275s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174310112 (107h11m39.771081126s) [seg_pos={0 0}, upload=119992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:11:42.174392163 (107h11m39.771163188s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:11:42.174399690 (107h11m39.771170715s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:12:02.252906846 (107h11m59.849677874s) [upload=119992]: Done uploading segments... Mar 10 22:12:02.253066497 (107h11m59.849837541s) [upload=119992]: Upload finished. err: metaclient: context canceled Duration: 47.616167178s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [............................................X.XXXXXXXXXXXXXX] duration: 5.001086972s trace: https://jaeger.dev.storj.io/trace/109982d58712e06c FAIL (2026-03-10 22:14:49 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001112239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 13.633139567s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:14:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 13.633186564s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.............................................X.XXXXXXXXXXXXX] duration: 5.000799666s trace: https://jaeger.dev.storj.io/trace/6f31a85b448bd1ba FAIL (2026-03-10 22:13:46 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00092544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.037200643s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:13:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 18.03722723s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..............................................X.XXXXXXXXXXXX] duration: 5.000861978s trace: https://jaeger.dev.storj.io/trace/d970e8f871278b5 FAIL (2026-03-10 22:13:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000884309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X..X..] duration: 1.38265302s trace: https://jaeger.dev.storj.io/trace/236bcdc90d857f3c FAIL (2026-03-10 22:13:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 10 22:11:38.970020774 (107h11m36.566791803s) [upload=119992]: Starting upload Mar 10 22:11:38.970116316 (107h11m36.566887331s) [upload=119992]: Done starting upload Mar 10 22:11:38.970424312 (107h11m36.567195326s) [upload=119992]: Uploading segments... Mar 10 22:11:38.970574854 (107h11m36.567345885s) [upload=119992]: Got next segment. Inline: false Mar 10 22:11:38.970650119 (107h11m36.567421134s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:38.970657466 (107h11m36.567428491s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:11:41.578785710 (107h11m39.175556741s) [seg_pos={0 0}, upload=119992]: Begin upload segment... Mar 10 22:11:41.578861456 (107h11m39.175632482s) [seg_pos={0 0}, upload=119992]: Done begin upload segment. Mar 10 22:11:41.578910683 (107h11m39.175681716s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.578926373 (107h11m39.175697402s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579585265 (107h11m39.176356295s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579908593 (107h11m39.176679619s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580190868 (107h11m39.176961906s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580310098 (107h11m39.17708112s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580644652 (107h11m39.177415681s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580658828 (107h11m39.177429864s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581041094 (107h11m39.177812122s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581099790 (107h11m39.177870825s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581480438 (107h11m39.178251469s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581532640 (107h11m39.178303668s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581870926 (107h11m39.178641952s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581963946 (107h11m39.178734979s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582201392 (107h11m39.178972424s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582405729 (107h11m39.179176767s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582870960 (107h11m39.179641996s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.583377134 (107h11m39.180148161s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.659615112 (107h11m39.256386141s) [upload=119992]: Next returned nil segment Mar 10 22:11:41.659705421 (107h11m39.256476449s) [upload=119992]: Waiting for error group managing segments... Mar 10 22:11:41.673205837 (107h11m39.269976868s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677131898 (107h11m39.273902921s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677231485 (107h11m39.274002513s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680107367 (107h11m39.276878391s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680874797 (107h11m39.277645826s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.684631539 (107h11m39.281402562s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.685306601 (107h11m39.282077629s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.707110296 (107h11m39.303881323s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.708508816 (107h11m39.305279832s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.779617247 (107h11m39.37638827s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.970373836 (107h11m39.567144863s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.129282623 (107h11m39.726053635s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.160167419 (107h11m39.756938442s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173664351 (107h11m39.770435374s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173688329 (107h11m39.770459353s) [seg_pos={0 0}, upload=119992]: Satisfied cohort requirements after 14 pieces Mar 10 22:11:42.173944215 (107h11m39.77071525s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.173954485 (107h11m39.770725509s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174094844 (107h11m39.770865874s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sK7oayGYBqB4fhjodVcZLid7438rnseaGQqTuZ4pTU5YNSuHi): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174110699 (107h11m39.770881729s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174127203 (107h11m39.770898235s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n739DCDptMWxwBZJ9sGaZLWbpYrCEFQRk8UDZe6QqrYmhGofF): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174133958 (107h11m39.770904981s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174283426 (107h11m39.771054454s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uH4iMUgUmHa22vDB92ERVX4G7PDch6T2zLw8ekGMiUxMiB3b2): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174291249 (107h11m39.771062275s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174310112 (107h11m39.771081126s) [seg_pos={0 0}, upload=119992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:11:42.174392163 (107h11m39.771163188s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:11:42.174399690 (107h11m39.771170715s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:12:02.252906846 (107h11m59.849677874s) [upload=119992]: Done uploading segments... Mar 10 22:12:02.253066497 (107h11m59.849837541s) [upload=119992]: Upload finished. err: metaclient: context canceled Duration: 1.382677326s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 48.611578862s trace: https://jaeger.dev.storj.io/trace/236bcdc90d857f3c FAIL (2026-03-10 22:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 10 22:11:38.970020774 (107h11m36.566791803s) [upload=119992]: Starting upload Mar 10 22:11:38.970116316 (107h11m36.566887331s) [upload=119992]: Done starting upload Mar 10 22:11:38.970424312 (107h11m36.567195326s) [upload=119992]: Uploading segments... Mar 10 22:11:38.970574854 (107h11m36.567345885s) [upload=119992]: Got next segment. Inline: false Mar 10 22:11:38.970650119 (107h11m36.567421134s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:38.970657466 (107h11m36.567428491s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:11:41.578785710 (107h11m39.175556741s) [seg_pos={0 0}, upload=119992]: Begin upload segment... Mar 10 22:11:41.578861456 (107h11m39.175632482s) [seg_pos={0 0}, upload=119992]: Done begin upload segment. Mar 10 22:11:41.578910683 (107h11m39.175681716s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.578926373 (107h11m39.175697402s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579585265 (107h11m39.176356295s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579908593 (107h11m39.176679619s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580190868 (107h11m39.176961906s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580310098 (107h11m39.17708112s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580644652 (107h11m39.177415681s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580658828 (107h11m39.177429864s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581041094 (107h11m39.177812122s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581099790 (107h11m39.177870825s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581480438 (107h11m39.178251469s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581532640 (107h11m39.178303668s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581870926 (107h11m39.178641952s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581963946 (107h11m39.178734979s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582201392 (107h11m39.178972424s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582405729 (107h11m39.179176767s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582870960 (107h11m39.179641996s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.583377134 (107h11m39.180148161s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.659615112 (107h11m39.256386141s) [upload=119992]: Next returned nil segment Mar 10 22:11:41.659705421 (107h11m39.256476449s) [upload=119992]: Waiting for error group managing segments... Mar 10 22:11:41.673205837 (107h11m39.269976868s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677131898 (107h11m39.273902921s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677231485 (107h11m39.274002513s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680107367 (107h11m39.276878391s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680874797 (107h11m39.277645826s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.684631539 (107h11m39.281402562s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.685306601 (107h11m39.282077629s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.707110296 (107h11m39.303881323s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.708508816 (107h11m39.305279832s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.779617247 (107h11m39.37638827s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.970373836 (107h11m39.567144863s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.129282623 (107h11m39.726053635s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.160167419 (107h11m39.756938442s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173664351 (107h11m39.770435374s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173688329 (107h11m39.770459353s) [seg_pos={0 0}, upload=119992]: Satisfied cohort requirements after 14 pieces Mar 10 22:11:42.173944215 (107h11m39.77071525s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.173954485 (107h11m39.770725509s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174094844 (107h11m39.770865874s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sK7oayGYBqB4fhjodVcZLid7438rnseaGQqTuZ4pTU5YNSuHi): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174110699 (107h11m39.770881729s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174127203 (107h11m39.770898235s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n739DCDptMWxwBZJ9sGaZLWbpYrCEFQRk8UDZe6QqrYmhGofF): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174133958 (107h11m39.770904981s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174283426 (107h11m39.771054454s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uH4iMUgUmHa22vDB92ERVX4G7PDch6T2zLw8ekGMiUxMiB3b2): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174291249 (107h11m39.771062275s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174310112 (107h11m39.771081126s) [seg_pos={0 0}, upload=119992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:11:42.174392163 (107h11m39.771163188s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:11:42.174399690 (107h11m39.771170715s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:12:02.252906846 (107h11m59.849677874s) [upload=119992]: Done uploading segments... Mar 10 22:12:02.253066497 (107h11m59.849837541s) [upload=119992]: Upload finished. err: metaclient: context canceled Duration: 48.611618435s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.084419422s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:12:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 15.084464112s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.788240627s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:12:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 8.78825732s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 23.319007732s trace: https://jaeger.dev.storj.io/trace/236bcdc90d857f3c FAIL (2026-03-10 22:12:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 10 22:11:38.970020774 (107h11m36.566791803s) [upload=119992]: Starting upload Mar 10 22:11:38.970116316 (107h11m36.566887331s) [upload=119992]: Done starting upload Mar 10 22:11:38.970424312 (107h11m36.567195326s) [upload=119992]: Uploading segments... Mar 10 22:11:38.970574854 (107h11m36.567345885s) [upload=119992]: Got next segment. Inline: false Mar 10 22:11:38.970650119 (107h11m36.567421134s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:38.970657466 (107h11m36.567428491s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:11:41.578785710 (107h11m39.175556741s) [seg_pos={0 0}, upload=119992]: Begin upload segment... Mar 10 22:11:41.578861456 (107h11m39.175632482s) [seg_pos={0 0}, upload=119992]: Done begin upload segment. Mar 10 22:11:41.578910683 (107h11m39.175681716s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.578926373 (107h11m39.175697402s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579585265 (107h11m39.176356295s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.579908593 (107h11m39.176679619s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580190868 (107h11m39.176961906s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580310098 (107h11m39.17708112s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580644652 (107h11m39.177415681s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.580658828 (107h11m39.177429864s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581041094 (107h11m39.177812122s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581099790 (107h11m39.177870825s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581480438 (107h11m39.178251469s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581532640 (107h11m39.178303668s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581870926 (107h11m39.178641952s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.581963946 (107h11m39.178734979s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582201392 (107h11m39.178972424s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582405729 (107h11m39.179176767s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.582870960 (107h11m39.179641996s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.583377134 (107h11m39.180148161s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Uploading piece... Mar 10 22:11:41.659615112 (107h11m39.256386141s) [upload=119992]: Next returned nil segment Mar 10 22:11:41.659705421 (107h11m39.256476449s) [upload=119992]: Waiting for error group managing segments... Mar 10 22:11:41.673205837 (107h11m39.269976868s) [noise=true , address=45.91.116.6:4980 , piece_id=NHCHW4RRVQO2PAWO3ENFSYXDMK2AVOAPU7X7DZAQAXXG42FZSJSQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677131898 (107h11m39.273902921s) [noise=true , address=45.91.116.13:4186 , piece_id=4YN6KL4WEIPAWPA6FZHLFAHTGUSDNVVLUTUVGAMB2BV2ADJ2X6CQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.677231485 (107h11m39.274002513s) [noise=true , address=45.91.116.5:4325 , piece_id=JFRZRFHFLFQPYCSNM5ONSPHU23M2YN7BXD2RWKOH2TQM6OP5Y75A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680107367 (107h11m39.276878391s) [noise=true , address=45.91.116.10:4100 , piece_id=VY7SBGU673LY3RS46XPN2WKAN6NJNUXFWQPBU6RUYSUR2TPKSMAA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.680874797 (107h11m39.277645826s) [noise=true , address=45.91.116.11:4135 , piece_id=N7TVDF2UXQC36VQSH5643AR3PFPHICDSU34TXIBPF2B5KTQAXUHA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.684631539 (107h11m39.281402562s) [noise=true , address=45.91.116.7:4029 , piece_id=3O3ELEULMNJ3CY2WSVD7DJJAPH6CMDF4IYPDGU263DGGA5V5SGSA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.685306601 (107h11m39.282077629s) [noise=true , address=79.127.219.38:4851 , piece_id=DXTOCCEPHSVIC33MAEDPIY4BSS2XHKVCQSBA5NJ27X7BT5ADP3FA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.707110296 (107h11m39.303881323s) [noise=true , address=77.90.31.21:4909 , piece_id=IKZFYOLMBLCKZBRKC66X3OVS5ZAAR253YTAN3YBMWT22BOPFM5IA, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.708508816 (107h11m39.305279832s) [noise=true , address=145.79.227.34:4927 , piece_id=SHLU75QFHNJ5DIIXIE4EGR73UPYVSMMJL2CYRG4PU6IVTWO53MBQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.779617247 (107h11m39.37638827s) [noise=true , address=145.79.227.3:4162 , piece_id=UVNORKXGF7ZVLJFBOWJICPZRNORZVIC4KOGZUIDYIL5GE25C5NVQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:41.970373836 (107h11m39.567144863s) [noise=true , address=167.17.72.70:4768 , piece_id=5XQGW4GEUMKD227OWYJM5BZEKM7U2QAV5TPMA4YDP2YTDFW7LJRQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.129282623 (107h11m39.726053635s) [noise=true , address=79.127.205.235:4267 , piece_id=PF3NHOQW7ANFK6DJNDXPX5M26RSWT5QAAH33OMNEV345U3VNVW5A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.160167419 (107h11m39.756938442s) [noise=true , address=79.127.205.234:4212 , piece_id=I3QNTHMRHW27MDUKH6GPJW6OS2EDNC3FDLCAALD7ODSMGLTKWLCQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173664351 (107h11m39.770435374s) [noise=true , address=193.149.164.70:4124 , piece_id=RQPS27LK7TLLJYQG5UYLLIA6GHFGESQU3JUXXMLCT6XU6MLI3MTQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: <nil> Mar 10 22:11:42.173688329 (107h11m39.770459353s) [seg_pos={0 0}, upload=119992]: Satisfied cohort requirements after 14 pieces Mar 10 22:11:42.173944215 (107h11m39.77071525s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.173954485 (107h11m39.770725509s) [noise=true , address=64.49.13.140:4506 , piece_id=E2G7A3IVDHXEOQKEK55H3WLGFPMPR63EJP2FLADDKBVASSVAMR6A, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174094844 (107h11m39.770865874s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sK7oayGYBqB4fhjodVcZLid7438rnseaGQqTuZ4pTU5YNSuHi): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174110699 (107h11m39.770881729s) [noise=true , address=193.149.164.66:4996 , piece_id=2PFKOK3MEX4I2NFBNKDVEJRPAOLCFN4DVXRORU3EAVHPEJCOUCZQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174127203 (107h11m39.770898235s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n739DCDptMWxwBZJ9sGaZLWbpYrCEFQRk8UDZe6QqrYmhGofF): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174133958 (107h11m39.770904981s) [noise=true , address=193.149.164.69:4085 , piece_id=SCOFFZCIVE5MIVRHSXB5VU2ODCQZSM65JXWVAMJLYUXM53GAR3HQ, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174283426 (107h11m39.771054454s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uH4iMUgUmHa22vDB92ERVX4G7PDch6T2zLw8ekGMiUxMiB3b2): protocol: expected piece hash; context canceled; EOF Mar 10 22:11:42.174291249 (107h11m39.771062275s) [noise=true , address=193.149.164.67:4011 , piece_id=AU6G324DUKZYAUASY2FRLI6ZTN5R3COB6VKFMFUGEMU2C5BUXJ6Q, seg_pos={0 0}, upload=119992]: Long tail cancellation due to optimal threshold Mar 10 22:11:42.174310112 (107h11m39.771081126s) [seg_pos={0 0}, upload=119992]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:11:42.174392163 (107h11m39.771163188s) [upload=119992]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:11:42.174399690 (107h11m39.771170715s) [upload=119992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:12:02.252906846 (107h11m59.849677874s) [upload=119992]: Done uploading segments... Mar 10 22:12:02.253066497 (107h11m59.849837541s) [upload=119992]: Upload finished. err: metaclient: context canceled Duration: 23.319028102s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001275187s trace: https://jaeger.dev.storj.io/trace/bf9e0416a3d8566 FAIL (2026-03-10 22:12:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:11:01.854662020 (107h10m59.451433045s) [upload=119986]: Starting upload Mar 10 22:11:01.854734670 (107h10m59.451505696s) [upload=119986]: Done starting upload Mar 10 22:11:01.854816376 (107h10m59.451587393s) [upload=119986]: Uploading segments... Mar 10 22:11:01.854860839 (107h10m59.451631868s) [upload=119986]: Got next segment. Inline: true Mar 10 22:11:01.854878692 (107h10m59.451649715s) [upload=119986]: Waiting for error group managing segments... Mar 10 22:11:01.854924815 (107h10m59.451695838s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:11:01.854930090 (107h10m59.451701113s) [upload=119986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 10 22:11:01.854933908 (107h10m59.451704932s) [upload=119986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:11:28.429373234 (107h11m26.026144248s) [upload=119986]: Done uploading segments... Mar 10 22:11:28.429393088 (107h11m26.026164109s) [upload=119986]: Upload finished. err: <nil> Duration:* 1m0.00137901s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...............................................X.XXXXXXXXXXX] duration: 5.001965919s trace: https://jaeger.dev.storj.io/trace/77b5add28f41e69 FAIL (2026-03-10 22:11:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001987535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 39.081696094s trace: https://jaeger.dev.storj.io/trace/77a488a17d559e93 FAIL (2026-03-10 22:11:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:09:19.958657469 (107h9m17.555428525s) [upload=119953]: Starting upload Mar 10 22:09:19.958728077 (107h9m17.555499112s) [upload=119953]: Done starting upload Mar 10 22:09:19.958862702 (107h9m17.555633725s) [upload=119953]: Uploading segments... Mar 10 22:09:19.959157921 (107h9m17.555928944s) [upload=119953]: Got next segment. Inline: false Mar 10 22:09:19.959213847 (107h9m17.55598487s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:09:19.959221365 (107h9m17.555992379s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:09:43.237428684 (107h9m40.834199708s) [seg_pos={0 0}, upload=119953]: Begin upload segment... Mar 10 22:09:43.237490392 (107h9m40.834261416s) [seg_pos={0 0}, upload=119953]: Done begin upload segment. Mar 10 22:09:43.237530197 (107h9m40.83430123s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237579310 (107h9m40.834350345s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237693153 (107h9m40.834464183s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238090436 (107h9m40.834861462s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238435160 (107h9m40.835206182s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238870843 (107h9m40.835641873s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239324982 (107h9m40.836096012s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239393496 (107h9m40.836164535s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240089813 (107h9m40.836860848s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240146176 (107h9m40.836917221s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240660980 (107h9m40.837432014s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240686781 (107h9m40.837457813s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241149498 (107h9m40.837920529s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241367745 (107h9m40.83813878s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241623017 (107h9m40.838394043s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242048046 (107h9m40.838819073s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242488756 (107h9m40.839259783s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242894369 (107h9m40.839665392s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.302150518 (107h9m40.89892154s) [upload=119953]: Next returned nil segment Mar 10 22:09:43.302163956 (107h9m40.89893498s) [upload=119953]: Waiting for error group managing segments... Mar 10 22:09:43.305668490 (107h9m40.902439525s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307283316 (107h9m40.904054346s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307517960 (107h9m40.904288986s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.311811458 (107h9m40.908582488s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.323266296 (107h9m40.920037319s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325820172 (107h9m40.922591206s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325871415 (107h9m40.922642438s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.330043241 (107h9m40.926814275s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.333647617 (107h9m40.930418641s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.334678420 (107h9m40.931449443s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.415586947 (107h9m41.012357961s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.616862910 (107h9m41.213633941s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.793747777 (107h9m41.390518796s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817813130 (107h9m41.414584153s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817854388 (107h9m41.414625415s) [seg_pos={0 0}, upload=119953]: Satisfied cohort requirements after 14 pieces Mar 10 22:09:43.818176974 (107h9m41.414948017s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16UoWyRRrvp9F8caokWXAKhCQg3KXaSKUzEu9g9xze7WPAEdxQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818193499 (107h9m41.414964533s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818319604 (107h9m41.415090637s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iyqAAQMR9JxyNWzTR1XR9hnUUdTfhdBidDparNqvPsDn3N7om): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818335973 (107h9m41.415107002s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818561902 (107h9m41.415332934s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818579221 (107h9m41.415350245s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818783992 (107h9m41.415555021s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgaSS1NScJhbWPu6uXvqGY3vzP1eADXR9k78UN8wYEVRZg7vQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818797525 (107h9m41.415568551s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818820039 (107h9m41.415591053s) [seg_pos={0 0}, upload=119953]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:09:43.818932738 (107h9m41.415703762s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:09:43.818949035 (107h9m41.415720058s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:09:43.875470641 (107h9m41.472241664s) [upload=119953]: Done uploading segments... Mar 10 22:09:43.875489369 (107h9m41.472260392s) [upload=119953]: Upload finished. err: <nil> Duration: 39.081718709s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [................................................X.XXXXXXXXXX] duration: 5.001301316s trace: https://jaeger.dev.storj.io/trace/429ebd51684fbfc1 FAIL (2026-03-10 22:11:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001328964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................................................X.XXXXXXXXX] duration: 5.001534121s trace: https://jaeger.dev.storj.io/trace/7278ee7c9f118a10 FAIL (2026-03-10 22:11:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001555689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 34.381630817s trace: https://jaeger.dev.storj.io/trace/77a488a17d559e93 FAIL (2026-03-10 22:10:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:09:19.958657469 (107h9m17.555428525s) [upload=119953]: Starting upload Mar 10 22:09:19.958728077 (107h9m17.555499112s) [upload=119953]: Done starting upload Mar 10 22:09:19.958862702 (107h9m17.555633725s) [upload=119953]: Uploading segments... Mar 10 22:09:19.959157921 (107h9m17.555928944s) [upload=119953]: Got next segment. Inline: false Mar 10 22:09:19.959213847 (107h9m17.55598487s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:09:19.959221365 (107h9m17.555992379s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:09:43.237428684 (107h9m40.834199708s) [seg_pos={0 0}, upload=119953]: Begin upload segment... Mar 10 22:09:43.237490392 (107h9m40.834261416s) [seg_pos={0 0}, upload=119953]: Done begin upload segment. Mar 10 22:09:43.237530197 (107h9m40.83430123s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237579310 (107h9m40.834350345s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237693153 (107h9m40.834464183s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238090436 (107h9m40.834861462s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238435160 (107h9m40.835206182s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238870843 (107h9m40.835641873s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239324982 (107h9m40.836096012s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239393496 (107h9m40.836164535s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240089813 (107h9m40.836860848s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240146176 (107h9m40.836917221s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240660980 (107h9m40.837432014s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240686781 (107h9m40.837457813s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241149498 (107h9m40.837920529s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241367745 (107h9m40.83813878s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241623017 (107h9m40.838394043s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242048046 (107h9m40.838819073s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242488756 (107h9m40.839259783s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242894369 (107h9m40.839665392s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.302150518 (107h9m40.89892154s) [upload=119953]: Next returned nil segment Mar 10 22:09:43.302163956 (107h9m40.89893498s) [upload=119953]: Waiting for error group managing segments... Mar 10 22:09:43.305668490 (107h9m40.902439525s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307283316 (107h9m40.904054346s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307517960 (107h9m40.904288986s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.311811458 (107h9m40.908582488s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.323266296 (107h9m40.920037319s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325820172 (107h9m40.922591206s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325871415 (107h9m40.922642438s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.330043241 (107h9m40.926814275s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.333647617 (107h9m40.930418641s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.334678420 (107h9m40.931449443s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.415586947 (107h9m41.012357961s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.616862910 (107h9m41.213633941s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.793747777 (107h9m41.390518796s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817813130 (107h9m41.414584153s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817854388 (107h9m41.414625415s) [seg_pos={0 0}, upload=119953]: Satisfied cohort requirements after 14 pieces Mar 10 22:09:43.818176974 (107h9m41.414948017s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16UoWyRRrvp9F8caokWXAKhCQg3KXaSKUzEu9g9xze7WPAEdxQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818193499 (107h9m41.414964533s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818319604 (107h9m41.415090637s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iyqAAQMR9JxyNWzTR1XR9hnUUdTfhdBidDparNqvPsDn3N7om): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818335973 (107h9m41.415107002s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818561902 (107h9m41.415332934s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818579221 (107h9m41.415350245s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818783992 (107h9m41.415555021s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgaSS1NScJhbWPu6uXvqGY3vzP1eADXR9k78UN8wYEVRZg7vQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818797525 (107h9m41.415568551s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818820039 (107h9m41.415591053s) [seg_pos={0 0}, upload=119953]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:09:43.818932738 (107h9m41.415703762s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:09:43.818949035 (107h9m41.415720058s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:09:43.875470641 (107h9m41.472241664s) [upload=119953]: Done uploading segments... Mar 10 22:09:43.875489369 (107h9m41.472260392s) [upload=119953]: Upload finished. err: <nil> Duration: 34.381650723s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................................................X.XXXXXXXX] duration: 5.000822557s trace: https://jaeger.dev.storj.io/trace/2c250957817104c6 FAIL (2026-03-10 22:10:36 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000845659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.006030645s trace: https://jaeger.dev.storj.io/trace/77a488a17d559e93 FAIL (2026-03-10 22:10:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 10 22:09:19.958657469 (107h9m17.555428525s) [upload=119953]: Starting upload Mar 10 22:09:19.958728077 (107h9m17.555499112s) [upload=119953]: Done starting upload Mar 10 22:09:19.958862702 (107h9m17.555633725s) [upload=119953]: Uploading segments... Mar 10 22:09:19.959157921 (107h9m17.555928944s) [upload=119953]: Got next segment. Inline: false Mar 10 22:09:19.959213847 (107h9m17.55598487s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 22:09:19.959221365 (107h9m17.555992379s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 22:09:43.237428684 (107h9m40.834199708s) [seg_pos={0 0}, upload=119953]: Begin upload segment... Mar 10 22:09:43.237490392 (107h9m40.834261416s) [seg_pos={0 0}, upload=119953]: Done begin upload segment. Mar 10 22:09:43.237530197 (107h9m40.83430123s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237579310 (107h9m40.834350345s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.237693153 (107h9m40.834464183s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238090436 (107h9m40.834861462s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238435160 (107h9m40.835206182s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.238870843 (107h9m40.835641873s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239324982 (107h9m40.836096012s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.239393496 (107h9m40.836164535s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240089813 (107h9m40.836860848s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240146176 (107h9m40.836917221s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240660980 (107h9m40.837432014s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.240686781 (107h9m40.837457813s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241149498 (107h9m40.837920529s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241367745 (107h9m40.83813878s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.241623017 (107h9m40.838394043s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242048046 (107h9m40.838819073s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242488756 (107h9m40.839259783s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.242894369 (107h9m40.839665392s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Uploading piece... Mar 10 22:09:43.302150518 (107h9m40.89892154s) [upload=119953]: Next returned nil segment Mar 10 22:09:43.302163956 (107h9m40.89893498s) [upload=119953]: Waiting for error group managing segments... Mar 10 22:09:43.305668490 (107h9m40.902439525s) [noise=true , address=79.127.219.46:4908 , piece_id=NARGDCJFJA5UQRWK2J5T7UMQAWAGFYWMLSX7V2PDX25K2CEWL5XA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307283316 (107h9m40.904054346s) [noise=true , address=79.127.219.38:4842 , piece_id=RROBSD26BSM76CMQD7THBQMPZBGTV3Z7NLCVOLOV2Q2HTLAPGCIQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.307517960 (107h9m40.904288986s) [noise=true , address=77.90.31.21:4882 , piece_id=JIF3XBXF3TWU4ATVMDNWY73PUYM4NHLX7AFJTSGJ5Z5IYRMREX4A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.311811458 (107h9m40.908582488s) [noise=true , address=145.79.227.19:4087 , piece_id=R4TM4MUEGPFZOZWDWAOZIBIK2XUI3PQ7G33M5NTK4L7BMZBAFHLQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.323266296 (107h9m40.920037319s) [noise=true , address=45.91.116.12:4235 , piece_id=R5PV2FHQV7PFBYODS2VL65XAX3PY52NEQ6HDUFMQYDSRRZ3NIJ4Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325820172 (107h9m40.922591206s) [noise=true , address=77.90.31.14:4696 , piece_id=6JZFHAFTASROJOQCIBCVZ4XBAVGY2LUWL33CXIMQFI6LTOJHQPRA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.325871415 (107h9m40.922642438s) [noise=true , address=145.79.227.17:4994 , piece_id=QNPXCQABENEALVLQH43623FQX4SYPWDNOGC4IDREPE5JI4VJBYMA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.330043241 (107h9m40.926814275s) [noise=true , address=77.90.31.11:4585 , piece_id=7WPYLKIMTT7E27UDJAMHO7M4IJ4BMVLVUNEHWQZ4IEW7OOUS4WCA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.333647617 (107h9m40.930418641s) [noise=true , address=145.79.227.31:4816 , piece_id=TETDQCAKFSCEVT3VYXNVI5GSXCPXODHNGUVI4NAH3JCR7CTKZTQA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.334678420 (107h9m40.931449443s) [noise=true , address=77.90.31.12:4606 , piece_id=LWNRREFYD4GKITNU26MLAKC2BB5J2MRX3IKCIC7RJL2BDHPD7YGA, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.415586947 (107h9m41.012357961s) [noise=true , address=45.91.116.5:4321 , piece_id=KQI3Z3NGU4H6GD6IR2SRTIBPXJJCWEMJCND4IP6AYWGL74UAGQAQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.616862910 (107h9m41.213633941s) [noise=true , address=167.17.72.69:4728 , piece_id=GATHRW55R7ZQRHJ7SKC7ZRHFHKOXHZAKJEACCBX3I2OOMFSDI6WQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.793747777 (107h9m41.390518796s) [noise=true , address=79.127.205.233:4170 , piece_id=DDOZXALVG5KRABZBYOXWEQS7CORG4CIKYGKLEUTRENPDOULDUHUQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817813130 (107h9m41.414584153s) [noise=true , address=79.127.205.235:4273 , piece_id=SNMOGGJIJ47ZNS6LERH6FC4QCPPJSTAVXNWWPUGSYRYVDLJVZOPQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: <nil> Mar 10 22:09:43.817854388 (107h9m41.414625415s) [seg_pos={0 0}, upload=119953]: Satisfied cohort requirements after 14 pieces Mar 10 22:09:43.818176974 (107h9m41.414948017s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16UoWyRRrvp9F8caokWXAKhCQg3KXaSKUzEu9g9xze7WPAEdxQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818193499 (107h9m41.414964533s) [noise=true , address=193.149.164.68:4054 , piece_id=V6KRMO6PXM7A45ZHKJH2VP3PXTXOL6TM4YFEYMQMYIE7LCNS2T6A, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818319604 (107h9m41.415090637s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iyqAAQMR9JxyNWzTR1XR9hnUUdTfhdBidDparNqvPsDn3N7om): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818335973 (107h9m41.415107002s) [noise=true , address=193.149.164.66:4973 , piece_id=RN6NBHSG5HLE4KLI23IFFV2OBZCYTSB54OFEYKOGD5C4AO24COEQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818561902 (107h9m41.415332934s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818579221 (107h9m41.415350245s) [noise=true , address=64.49.13.27:4968 , piece_id=WVO45VYERUKVBTTMYVEYB72VWZNJ75MNGPUBUBTRI2SJ4PTG4X7Q, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818783992 (107h9m41.415555021s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TgaSS1NScJhbWPu6uXvqGY3vzP1eADXR9k78UN8wYEVRZg7vQ): protocol: expected piece hash; context canceled; EOF Mar 10 22:09:43.818797525 (107h9m41.415568551s) [noise=true , address=193.149.164.71:4291 , piece_id=YC5AXORVAO6WWWZZDMHZV556WH3MPRUXTJJC73RYQ7T2HKV7D6SQ, seg_pos={0 0}, upload=119953]: Long tail cancellation due to optimal threshold Mar 10 22:09:43.818820039 (107h9m41.415591053s) [seg_pos={0 0}, upload=119953]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Mar 10 22:09:43.818932738 (107h9m41.415703762s) [upload=119953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 10 22:09:43.818949035 (107h9m41.415720058s) [upload=119953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 10 22:09:43.875470641 (107h9m41.472241664s) [upload=119953]: Done uploading segments... Mar 10 22:09:43.875489369 (107h9m41.472260392s) [upload=119953]: Upload finished. err: <nil> Duration: 1m0.006060105s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................................................X.XXXXXXX] duration: 5.001512069s trace: https://jaeger.dev.storj.io/trace/57e633c16ccdfcbc FAIL (2026-03-10 22:09:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00153989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....................................................X.XXXXXX] duration: 5.001021713s trace: https://jaeger.dev.storj.io/trace/762d08d400431fef FAIL (2026-03-10 22:08:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00103329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....................................................X.XXXXX] duration: 5.001811796s trace: https://jaeger.dev.storj.io/trace/1dafee6440a580e6 FAIL (2026-03-10 22:07:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00197951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................................................X.XXXX] duration: 5.000733387s trace: https://jaeger.dev.storj.io/trace/2f7fc423e1d2278b FAIL (2026-03-10 22:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000772652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................................................X.XXX] duration: 5.001271092s trace: https://jaeger.dev.storj.io/trace/6baf0d050388ca30 FAIL (2026-03-10 22:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001302452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.......................................................X.XX] duration: 5.00172739s trace: https://jaeger.dev.storj.io/trace/c5e7e13dfc8ba00 FAIL (2026-03-10 22:05:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001818578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XX.......................................................X.X] duration: 5.000668754s trace: https://jaeger.dev.storj.io/trace/7dbb5c6ba295705f FAIL (2026-03-10 22:04:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000693167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XX.......................................................X.] duration: 4.487400606s trace: https://jaeger.dev.storj.io/trace/556fef404e52a685 FAIL (2026-03-10 22:04:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.487426736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XX.......................................................X] duration: 5.001383414s trace: https://jaeger.dev.storj.io/trace/556fef404e52a685 FAIL (2026-03-10 22:03:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001403114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...XX.......................................................] duration: 894.013537ms trace: https://jaeger.dev.storj.io/trace/698fd4ac72ce0f8b FAIL (2026-03-10 22:02:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 894.038961ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....XX......................................................] duration: 555.594204ms trace: https://jaeger.dev.storj.io/trace/698fd4ac72ce0f8b FAIL (2026-03-10 22:01:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.621987ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....XX.....................................................] duration: 598.247019ms trace: https://jaeger.dev.storj.io/trace/698fd4ac72ce0f8b FAIL (2026-03-10 22:00:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 598.298554ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......XX....................................................] duration: 603.85818ms trace: https://jaeger.dev.storj.io/trace/698fd4ac72ce0f8b FAIL (2026-03-10 21:58:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.887256ms core uplink us-select-1 us1-placement-20

1
...
386
387
388
389
390
...
6407