Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXX] duration: 5.00112518s trace: https://jaeger.dev.storj.io/trace/4fa3f745977a9fc5 FAIL (2026-03-18 21:52:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001157564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 33.202000479s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:52:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 33.202031456s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............................................XXXXXX.....XXXX] duration: 30.003804016s trace: https://jaeger.dev.storj.io/trace/409ead75bca4e1ef FAIL (2026-03-18 21:51:41 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:51:11.751282297 (298h51m9.348053309s) [upload=334725]: Starting upload Mar 18 21:51:11.751444687 (298h51m9.348215721s) [upload=334725]: Done starting upload Mar 18 21:51:11.751561345 (298h51m9.34833237s) [upload=334725]: Uploading segments... Mar 18 21:51:11.752004879 (298h51m9.348775904s) [upload=334725]: Got next segment. Inline: false Mar 18 21:51:11.752071954 (298h51m9.348842989s) [upload=334725]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:51:11.752079617 (298h51m9.348850646s) [upload=334725]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:51:15.958797944 (298h51m13.555568966s) [seg_pos={0 0}, upload=334725]: Begin upload segment... Mar 18 21:51:15.958852004 (298h51m13.555623017s) [seg_pos={0 0}, upload=334725]: Done begin upload segment. Mar 18 21:51:15.958894241 (298h51m13.555665267s) [noise=true , address=145.79.227.24:4584 , piece_id=PKMWBETQ7VHNVKRM6HWTNWZERAAFAMAGJERSX6FAKYSFNHSS2P2A, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.958910587 (298h51m13.555681621s) [noise=true , address=145.79.227.25:4619 , piece_id=J4I2VBSMDEIGLNOWGJFBBCY73KSLYIPWVHK552MMEYMC6O2DZLTA, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.959710997 (298h51m13.556482021s) [noise=true , address=145.79.227.48:4301 , piece_id=5JBG7DOKIPOWN33AV3KTJRGI4BCCBDITUP52PPLE5YAL7TDNJLEQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.959974957 (298h51m13.556745993s) [noise=true , address=77.90.31.12:4589 , piece_id=EJYRRHWSANWNMMG2KFHBSFY2JHVD67YMWIBYA6OJXRDOZ37O7QJQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.960126205 (298h51m13.556897228s) [noise=true , address=45.91.116.10:4081 , piece_id=USNUXPSWAEHC7ZIZFYICE5EYKNRFIPE32CXT7BCKNDIOJZ3CTVSQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.960476866 (298h51m13.5572479s) [noise=true , address=145.79.227.13:4807 , piece_id=JB3MTGBSZFQP4QKPM2HWWPXON6BZNOHFQHWPBF2PLZF7J4ZVTFFQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.960865291 (298h51m13.557636317s) [noise=true , address=45.91.116.12:4228 , piece_id=3KEI5446T6V4AXLB2REOS6F3HJDKSRSSNF22L4MMSLJAQYRM2BYQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.961250976 (298h51m13.558021999s) [noise=true , address=45.91.116.5:4299 , piece_id=IGEN5LFVWFDYRKXM2V2G6V3MQ2DGFTZTBI67B4TDHRFIEXWPBZ5Q, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.961427474 (298h51m13.558198507s) [noise=true , address=145.79.227.8:4331 , piece_id=K43K726RAERSFBAGUBUGVGH6TLEKSUMIJBF3KI6ZTR5ZJXXRLZXA, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.962130818 (298h51m13.558901849s) [noise=true , address=145.79.227.6:4262 , piece_id=HSB6HNL3XBFWN7GT45BP5T3HCH4FWOXMTOAWESBFCJBYOFX5WZXQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.962505269 (298h51m13.559276295s) [noise=true , address=145.79.227.35:4485 , piece_id=5WD22N7IUT36NEY3GDOI6UJXPKDT6JVHD4RSVR7QTWJEDTJVZ2HQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.962921296 (298h51m13.559692325s) [noise=true , address=77.90.31.19:4841 , piece_id=J53QWRVA4W4SL76GKJ2WAWQRONNMGFHFMZ6ZUCT3I4IQKE6VG4SQ, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.963427472 (298h51m13.560198495s) [noise=true , address=145.79.227.49:4339 , piece_id=BR5GPWNGTATC2IAJ3QHDR4UE5VNYAAZLI2IUFPDNAPARU6SG325Q, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.963888305 (298h51m13.560659335s) [noise=true , address=145.79.227.52:4445 , piece_id=KSB5HPC5SGHC7CNQV2QTBPY2HT65EMSMYW5T2YDO4RXZETB6HTDA, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.964277953 (298h51m13.561048976s) [noise=true , address=145.79.227.33:4870 , piece_id=HOX3LB5FK5MSEUFSV2ADPSSOHGGYFG7NMY767WPEQIGU4JT54OLA, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:15.964431111 (298h51m13.561202143s) [noise=true , address=145.79.227.12:4770 , piece_id=YZJPNVDJSVBAM267EP6RFN7VIZP2V7WCDJEFEXUE24XZNTU6I3HA, seg_pos={0 0}, upload=334725]: Uploading piece... Mar 18 21:51:16.031214739 (298h51m13.627985752s) [upload=334725]: Next returned nil segment Mar 18 21:51:16.031248816 (298h51m13.62801984s) [upload=334725]: Waiting for error group managing segments... Mar 18 21:51:16.037524922 (298h51m13.634295945s) [noise=true , address=45.91.116.10:4081 , piece_id=USNUXPSWAEHC7ZIZFYICE5EYKNRFIPE32CXT7BCKNDIOJZ3CTVSQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.039293198 (298h51m13.636064221s) [noise=true , address=145.79.227.48:4301 , piece_id=5JBG7DOKIPOWN33AV3KTJRGI4BCCBDITUP52PPLE5YAL7TDNJLEQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.039340026 (298h51m13.636111039s) [noise=true , address=145.79.227.35:4485 , piece_id=5WD22N7IUT36NEY3GDOI6UJXPKDT6JVHD4RSVR7QTWJEDTJVZ2HQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.040118869 (298h51m13.636889892s) [noise=true , address=45.91.116.5:4299 , piece_id=IGEN5LFVWFDYRKXM2V2G6V3MQ2DGFTZTBI67B4TDHRFIEXWPBZ5Q, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.040328076 (298h51m13.637099099s) [noise=true , address=77.90.31.19:4841 , piece_id=J53QWRVA4W4SL76GKJ2WAWQRONNMGFHFMZ6ZUCT3I4IQKE6VG4SQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.041087463 (298h51m13.637858485s) [noise=true , address=145.79.227.25:4619 , piece_id=J4I2VBSMDEIGLNOWGJFBBCY73KSLYIPWVHK552MMEYMC6O2DZLTA, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.041629417 (298h51m13.638400441s) [noise=true , address=145.79.227.6:4262 , piece_id=HSB6HNL3XBFWN7GT45BP5T3HCH4FWOXMTOAWESBFCJBYOFX5WZXQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.042147194 (298h51m13.638918217s) [noise=true , address=145.79.227.12:4770 , piece_id=YZJPNVDJSVBAM267EP6RFN7VIZP2V7WCDJEFEXUE24XZNTU6I3HA, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.042561910 (298h51m13.639332933s) [noise=true , address=145.79.227.33:4870 , piece_id=HOX3LB5FK5MSEUFSV2ADPSSOHGGYFG7NMY767WPEQIGU4JT54OLA, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.043048287 (298h51m13.63981931s) [noise=true , address=145.79.227.49:4339 , piece_id=BR5GPWNGTATC2IAJ3QHDR4UE5VNYAAZLI2IUFPDNAPARU6SG325Q, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.043120210 (298h51m13.639891233s) [noise=true , address=145.79.227.52:4445 , piece_id=KSB5HPC5SGHC7CNQV2QTBPY2HT65EMSMYW5T2YDO4RXZETB6HTDA, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.044123257 (298h51m13.64089428s) [noise=true , address=145.79.227.8:4331 , piece_id=K43K726RAERSFBAGUBUGVGH6TLEKSUMIJBF3KI6ZTR5ZJXXRLZXA, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.044422484 (298h51m13.641193522s) [noise=true , address=77.90.31.12:4589 , piece_id=EJYRRHWSANWNMMG2KFHBSFY2JHVD67YMWIBYA6OJXRDOZ37O7QJQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.044442680 (298h51m13.641213696s) [seg_pos={0 0}, upload=334725]: Satisfied cohort requirements after 13 pieces Mar 18 21:51:16.044606196 (298h51m13.64137722s) [noise=true , address=145.79.227.13:4807 , piece_id=JB3MTGBSZFQP4QKPM2HWWPXON6BZNOHFQHWPBF2PLZF7J4ZVTFFQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: <nil> Mar 18 21:51:16.044617546 (298h51m13.641388568s) [seg_pos={0 0}, upload=334725]: Satisfied cohort requirements after 14 pieces Mar 18 21:51:16.044829577 (298h51m13.64160061s) [noise=true , address=145.79.227.24:4584 , piece_id=PKMWBETQ7VHNVKRM6HWTNWZERAAFAMAGJERSX6FAKYSFNHSS2P2A, seg_pos={0 0}, upload=334725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TMMXMjWNxBgccLQMbuExNxgwvzQpNH4cizjaPZDufcjvTqdeh): protocol: expected piece hash; context canceled; EOF Mar 18 21:51:16.044836127 (298h51m13.641607167s) [noise=true , address=45.91.116.12:4228 , piece_id=3KEI5446T6V4AXLB2REOS6F3HJDKSRSSNF22L4MMSLJAQYRM2BYQ, seg_pos={0 0}, upload=334725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Mar 18 21:51:16.044850382 (298h51m13.641621415s) [noise=true , address=45.91.116.12:4228 , piece_id=3KEI5446T6V4AXLB2REOS6F3HJDKSRSSNF22L4MMSLJAQYRM2BYQ, seg_pos={0 0}, upload=334725]: Long tail cancellation due to optimal threshold Mar 18 21:51:16.044887138 (298h51m13.64165815s) [noise=true , address=145.79.227.24:4584 , piece_id=PKMWBETQ7VHNVKRM6HWTNWZERAAFAMAGJERSX6FAKYSFNHSS2P2A, seg_pos={0 0}, upload=334725]: Long tail cancellation due to optimal threshold Mar 18 21:51:16.044909648 (298h51m13.641680658s) [seg_pos={0 0}, upload=334725]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Mar 18 21:51:16.044996811 (298h51m13.641767841s) [upload=334725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:51:16.045002379 (298h51m13.641773444s) [upload=334725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:36.646156203 (298h51m34.242927227s) [upload=334725]: Done uploading segments... Mar 18 21:51:36.646189492 (298h51m34.242960515s) [upload=334725]: Upload finished. err: <nil> Duration: 30.003819674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 10 out of 60 is failed (threshold=3) [........................................X......XXXXXX.X...XX] duration: 30.001918009s trace: https://jaeger.dev.storj.io/trace/44a75dd33a5a1245 FAIL (2026-03-18 21:51:30 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:51:00.341900388 (298h50m57.938671422s) [upload=334720]: Starting upload Mar 18 21:51:00.341994120 (298h50m57.938765153s) [upload=334720]: Done starting upload Mar 18 21:51:00.342115053 (298h50m57.938886078s) [upload=334720]: Uploading segments... Mar 18 21:51:00.342189100 (298h50m57.938960128s) [upload=334720]: Got next segment. Inline: true Mar 18 21:51:00.342195821 (298h50m57.938966848s) [upload=334720]: Waiting for error group managing segments... Mar 18 21:51:00.342248065 (298h50m57.939019091s) [upload=334720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:51:00.342264864 (298h50m57.939035891s) [upload=334720]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:51:00.342268566 (298h50m57.939039589s) [upload=334720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:25.423105885 (298h51m23.019876923s) [upload=334720]: Done uploading segments... Mar 18 21:51:25.423162314 (298h51m23.019933347s) [upload=334720]: Upload finished. err: <nil> Duration:* 30.001945819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 24.993027921s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:51:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 24.993055467s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXX] duration: 5.001260851s trace: https://jaeger.dev.storj.io/trace/58399ff957f170e0 FAIL (2026-03-18 21:51:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001286843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........................................XX.....X.XXXXXX.X..X] duration: 30.006867793s trace: https://jaeger.dev.storj.io/trace/5ef3c9674a60f04 FAIL (2026-03-18 21:51:24 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006889106s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [......................................X....X...XXX..XX.X..X.] duration: 24.927569367s trace: https://jaeger.dev.storj.io/trace/3817074bbd799a98 FAIL (2026-03-18 21:51:23 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 24.927606069s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............................................XXXXXX.....XXX] duration: 30.024847211s trace: https://jaeger.dev.storj.io/trace/3a4f48b8fd788934 FAIL (2026-03-18 21:51:11 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:41.670332989 (298h50m39.267104004s) [upload=334714]: Starting upload Mar 18 21:50:41.670425680 (298h50m39.267196704s) [upload=334714]: Done starting upload Mar 18 21:50:41.670536980 (298h50m39.267307996s) [upload=334714]: Uploading segments... Mar 18 21:50:41.670873115 (298h50m39.267644147s) [upload=334714]: Got next segment. Inline: false Mar 18 21:50:41.670962746 (298h50m39.267733772s) [upload=334714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:41.670969711 (298h50m39.267740734s) [upload=334714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:50:57.938936122 (298h50m55.535707159s) [seg_pos={0 0}, upload=334714]: Begin upload segment... Mar 18 21:50:57.939009107 (298h50m55.535780141s) [seg_pos={0 0}, upload=334714]: Done begin upload segment. Mar 18 21:50:57.939094357 (298h50m55.535865389s) [noise=true , address=145.79.227.41:4086 , piece_id=WGJWLVIW3FYCKKVXZ3DJ4PEQKEOQXQT3JSAJTK22ZVY2SFBWAOHQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.939241873 (298h50m55.536012905s) [noise=true , address=145.79.227.52:4448 , piece_id=4TVE2THWMKWSKN7FPWPXPERFDC4S5JHA5YSL43MDNU5QYOWOQOQA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.940085856 (298h50m55.536856887s) [noise=true , address=145.79.227.21:4476 , piece_id=HLAZ5V562MFWCEQZ3LPJOS2BYWDSVTA7ZTEKZ7NVTALLP4DN2JPQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.940400649 (298h50m55.537171673s) [noise=true , address=145.79.227.50:4375 , piece_id=M25ADJC6DQ7AV6JYFVVG5BVFIQP2J3ZMOS2FTZZXWXMWHC6WI3JA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.940619019 (298h50m55.537390053s) [noise=true , address=145.79.227.43:4157 , piece_id=ZREWZ46EVKJVQOQLEHPPURKTDSEKDRDJ4673PEVQM33UMXW3FABA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.941055317 (298h50m55.537826357s) [noise=true , address=77.90.31.15:4701 , piece_id=N2FQ6HUWID7LEV3IYNL6ACO7KF6YGPRXNV4JZ3FTJPEMWWAQHSGA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.941092483 (298h50m55.537863514s) [noise=true , address=77.90.31.10:4522 , piece_id=ZIGK7CRQHFABFTKAN4SJOFINLAMGJ5TO73NWFBJTGF2XYQBZWOAQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.941554577 (298h50m55.538325601s) [noise=true , address=145.79.227.29:4946 , piece_id=TLJYVANQHG77DVHGXFNFPZ3ZTDNPG2TCCZ243ZHBHUWVSBGO4Y4A, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.941660522 (298h50m55.538431555s) [noise=true , address=79.127.219.38:4820 , piece_id=XHBFT7UXSODQNORY7SKYE3HQK3XOCZG3W7RQW4SVIUQ6KWBPX3MQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.942069126 (298h50m55.538840156s) [noise=true , address=145.79.227.22:4512 , piece_id=32GUB6T23JCTOL5LXHG7PE3BXIQSTNY5IZQBNUNCTJOEUMEUGGLQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.942190976 (298h50m55.538962002s) [noise=true , address=145.79.227.37:4571 , piece_id=VKA2EFEJ2GIJHTA4TBLVJAELJEMWCOEVYTKJPRJYYDNBTFLH62OA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.943133698 (298h50m55.539904731s) [noise=true , address=145.79.227.28:4728 , piece_id=MLWMJVFOHOHF2AXBTDN5B43JNODU2H5VWLVTRBKQAID57NSNVMBQ, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.943480055 (298h50m55.540251089s) [noise=true , address=145.79.227.20:4440 , piece_id=SKJBR5HMRSJI6NK4D6YJA5ODCSCU2NLQNY2MOJRZXQZCE35SJ6QA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.945521988 (298h50m55.542293021s) [noise=true , address=77.90.31.11:4558 , piece_id=MJM2BAHIWX7FQF7H4F37B342UCVITXJJ7EBXDQZSERVXVEC6JD3Q, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.946802650 (298h50m55.543573679s) [noise=true , address=145.79.227.34:4908 , piece_id=6JE2SY6W55UPHYUD35ZKAVM6IO33444JWY2M6ZBTJKKMLBUA355A, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:57.948787637 (298h50m55.54555867s) [noise=true , address=77.90.31.13:4627 , piece_id=G7FCY325CYRWQ3SHZJTQ2M2XFJTA7EOWWCRPUZT5HHFOADF2OBQA, seg_pos={0 0}, upload=334714]: Uploading piece... Mar 18 21:50:58.033699450 (298h50m55.63047048s) [upload=334714]: Next returned nil segment Mar 18 21:50:58.033717013 (298h50m55.630488046s) [upload=334714]: Waiting for error group managing segments... Mar 18 21:50:58.039894451 (298h50m55.636665475s) [noise=true , address=77.90.31.11:4558 , piece_id=MJM2BAHIWX7FQF7H4F37B342UCVITXJJ7EBXDQZSERVXVEC6JD3Q, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.041328754 (298h50m55.63809978s) [noise=true , address=77.90.31.10:4522 , piece_id=ZIGK7CRQHFABFTKAN4SJOFINLAMGJ5TO73NWFBJTGF2XYQBZWOAQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.044514478 (298h50m55.641285504s) [noise=true , address=79.127.219.38:4820 , piece_id=XHBFT7UXSODQNORY7SKYE3HQK3XOCZG3W7RQW4SVIUQ6KWBPX3MQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.044655695 (298h50m55.641426732s) [noise=true , address=77.90.31.13:4627 , piece_id=G7FCY325CYRWQ3SHZJTQ2M2XFJTA7EOWWCRPUZT5HHFOADF2OBQA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.045097868 (298h50m55.641868892s) [noise=true , address=77.90.31.15:4701 , piece_id=N2FQ6HUWID7LEV3IYNL6ACO7KF6YGPRXNV4JZ3FTJPEMWWAQHSGA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.045629093 (298h50m55.642400119s) [noise=true , address=145.79.227.21:4476 , piece_id=HLAZ5V562MFWCEQZ3LPJOS2BYWDSVTA7ZTEKZ7NVTALLP4DN2JPQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.045984309 (298h50m55.642755344s) [noise=true , address=145.79.227.52:4448 , piece_id=4TVE2THWMKWSKN7FPWPXPERFDC4S5JHA5YSL43MDNU5QYOWOQOQA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.047243170 (298h50m55.644014194s) [noise=true , address=145.79.227.22:4512 , piece_id=32GUB6T23JCTOL5LXHG7PE3BXIQSTNY5IZQBNUNCTJOEUMEUGGLQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.048365215 (298h50m55.645136244s) [noise=true , address=145.79.227.20:4440 , piece_id=SKJBR5HMRSJI6NK4D6YJA5ODCSCU2NLQNY2MOJRZXQZCE35SJ6QA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.048574064 (298h50m55.645345087s) [noise=true , address=145.79.227.34:4908 , piece_id=6JE2SY6W55UPHYUD35ZKAVM6IO33444JWY2M6ZBTJKKMLBUA355A, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.051532925 (298h50m55.648303956s) [noise=true , address=145.79.227.50:4375 , piece_id=M25ADJC6DQ7AV6JYFVVG5BVFIQP2J3ZMOS2FTZZXWXMWHC6WI3JA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.052412071 (298h50m55.649183122s) [noise=true , address=145.79.227.29:4946 , piece_id=TLJYVANQHG77DVHGXFNFPZ3ZTDNPG2TCCZ243ZHBHUWVSBGO4Y4A, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.057766661 (298h50m55.654537693s) [noise=true , address=145.79.227.28:4728 , piece_id=MLWMJVFOHOHF2AXBTDN5B43JNODU2H5VWLVTRBKQAID57NSNVMBQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: <nil> Mar 18 21:50:58.057793428 (298h50m55.654564462s) [seg_pos={0 0}, upload=334714]: Satisfied cohort requirements after 13 pieces Mar 18 21:50:58.058061945 (298h50m55.654832979s) [noise=true , address=145.79.227.37:4571 , piece_id=VKA2EFEJ2GIJHTA4TBLVJAELJEMWCOEVYTKJPRJYYDNBTFLH62OA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:58.058075488 (298h50m55.654846517s) [noise=true , address=145.79.227.37:4571 , piece_id=VKA2EFEJ2GIJHTA4TBLVJAELJEMWCOEVYTKJPRJYYDNBTFLH62OA, seg_pos={0 0}, upload=334714]: Long tail cancellation due to optimal threshold Mar 18 21:50:58.058506096 (298h50m55.655277128s) [noise=true , address=145.79.227.41:4086 , piece_id=WGJWLVIW3FYCKKVXZ3DJ4PEQKEOQXQT3JSAJTK22ZVY2SFBWAOHQ, seg_pos={0 0}, upload=334714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:58.058519906 (298h50m55.65529094s) [noise=true , address=145.79.227.41:4086 , piece_id=WGJWLVIW3FYCKKVXZ3DJ4PEQKEOQXQT3JSAJTK22ZVY2SFBWAOHQ, seg_pos={0 0}, upload=334714]: Long tail cancellation due to optimal threshold Mar 18 21:50:58.058707731 (298h50m55.655478765s) [noise=true , address=145.79.227.43:4157 , piece_id=ZREWZ46EVKJVQOQLEHPPURKTDSEKDRDJ4673PEVQM33UMXW3FABA, seg_pos={0 0}, upload=334714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:58.058716532 (298h50m55.655487566s) [noise=true , address=145.79.227.43:4157 , piece_id=ZREWZ46EVKJVQOQLEHPPURKTDSEKDRDJ4673PEVQM33UMXW3FABA, seg_pos={0 0}, upload=334714]: Long tail cancellation due to optimal threshold Mar 18 21:50:58.058743679 (298h50m55.655514695s) [seg_pos={0 0}, upload=334714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:50:58.058827717 (298h50m55.655598741s) [upload=334714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:50:58.058833973 (298h50m55.655604996s) [upload=334714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:11.659819680 (298h51m9.25659071s) [upload=334714]: Done uploading segments... Mar 18 21:51:11.659924639 (298h51m9.25669567s) [upload=334714]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.024876023s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXXX] duration: 5.00171678s trace: https://jaeger.dev.storj.io/trace/696a63821d6ef3c2 FAIL (2026-03-18 21:50:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001753063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 33.341302783s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:50:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 33.341326385s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.........................................X......XXXXXX.X...X] duration: 30.001159809s trace: https://jaeger.dev.storj.io/trace/721747c47c2fffc0 FAIL (2026-03-18 21:50:49 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:19.780632815 (298h50m17.37740383s) [upload=334708]: Starting upload Mar 18 21:50:19.780698594 (298h50m17.377469626s) [upload=334708]: Done starting upload Mar 18 21:50:19.780817554 (298h50m17.377588579s) [upload=334708]: Uploading segments... Mar 18 21:50:19.780873073 (298h50m17.377644104s) [upload=334708]: Got next segment. Inline: true Mar 18 21:50:19.780878906 (298h50m17.377649933s) [upload=334708]: Waiting for error group managing segments... Mar 18 21:50:19.780942800 (298h50m17.377713823s) [upload=334708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:19.780948358 (298h50m17.377719371s) [upload=334708]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:19.780952076 (298h50m17.377723099s) [upload=334708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:50:49.742803566 (298h50m47.339574638s) [upload=334708]: Done uploading segments... Mar 18 21:50:49.742881269 (298h50m47.339652302s) [upload=334708]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001219784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.......................................X....X...XXX..XX.X..X] duration: 30.001088629s trace: https://jaeger.dev.storj.io/trace/3817074bbd799a98 FAIL (2026-03-18 21:50:48 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001110811s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................................XX.....X.XXXXXX.X..] duration: 26.464018075s trace: https://jaeger.dev.storj.io/trace/2fb1108aa58b9f75 FAIL (2026-03-18 21:50:47 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.464051612s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XXXXXX.....XX] duration: 30.033904477s trace: https://jaeger.dev.storj.io/trace/699e148c7e7cf231 FAIL (2026-03-18 21:50:41 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:11.457336413 (298h50m9.054107447s) [upload=334705]: Starting upload Mar 18 21:50:11.457460053 (298h50m9.054231085s) [upload=334705]: Done starting upload Mar 18 21:50:11.457565524 (298h50m9.054336546s) [upload=334705]: Uploading segments... Mar 18 21:50:11.457966286 (298h50m9.054737304s) [upload=334705]: Got next segment. Inline: false Mar 18 21:50:11.458127974 (298h50m9.054899006s) [upload=334705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:11.458136231 (298h50m9.054907262s) [upload=334705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:50:27.757642832 (298h50m25.35441386s) [seg_pos={0 0}, upload=334705]: Begin upload segment... Mar 18 21:50:27.757738205 (298h50m25.354509237s) [seg_pos={0 0}, upload=334705]: Done begin upload segment. Mar 18 21:50:27.757777503 (298h50m25.354548537s) [noise=true , address=145.79.227.42:4124 , piece_id=KPLOTO2MU3TCFEWQGBLVC2FXGFHNYZG5Y4AJGPGI2Z6YUSDAWZ5A, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.757841975 (298h50m25.35461301s) [noise=true , address=145.79.227.40:4055 , piece_id=LXUZUGBSW6W6ZFRBMB743PWQQSMBXOENK77NZIXGRQZYVU3UPWIQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.758764191 (298h50m25.355535226s) [noise=true , address=145.79.227.2:4113 , piece_id=LDADQAB7RHDTGQNCGVXUFWC4L5552LGT75M6F2BT763SO22H7VMQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.758984359 (298h50m25.355755393s) [noise=true , address=145.79.227.31:4798 , piece_id=QDFKAJBFBBISOMGW5OEAOJFDVUALWPK46F33RZWVB7DGT2F2HMEA, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.759797080 (298h50m25.356568113s) [noise=true , address=145.79.227.27:4694 , piece_id=4EIWTOHMJJ6Z23OZBHLCG2NBNL55UR4PS5JDYB3SWAGFCZRPWSYA, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.763547588 (298h50m25.360318622s) [noise=true , address=77.90.31.17:4774 , piece_id=EEMBISWBKYNHPN6XNTQNLHF5E74T4JMORFFFOWR7VQMGSYQEHG7Q, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.764027997 (298h50m25.36079903s) [noise=true , address=145.79.227.16:4913 , piece_id=OQY2XYXGFI3WSELTXNDHTYXB57BJF2BDMPKOL2JUXUPC6N66A7NQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.764575425 (298h50m25.361346451s) [noise=true , address=145.79.227.39:4019 , piece_id=TWN5U4ZM5WUFFDFXPY5PUEAEIWCC5LIKJNJBMELFHMYYL3DC3CPA, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.764820676 (298h50m25.361591709s) [noise=true , address=45.91.116.7:4009 , piece_id=L52PA4RFKPQX4IIDFY2C3QDAPSRURKN5YQQCTKEPYBHBIBSOMX3Q, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.765507174 (298h50m25.362278208s) [noise=true , address=145.79.227.3:4153 , piece_id=3T3DCDSKUUM3DSZHGP5KSC3FOY2NLYDPELOUPFUCWYWMAPOFP3UQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.765539407 (298h50m25.362310443s) [noise=true , address=145.79.227.19:4081 , piece_id=6CCCUIQ7P67YLXPLTLLRKEZFW644G5KARTOILJGZ65VBFGNLNKTQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.766333329 (298h50m25.363104363s) [noise=true , address=77.90.31.15:4697 , piece_id=4HIKVKB3PRE6VQ74UDXU7EKHIVJ3RMPJ457HHKB7RLFCLU223OZQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.766456885 (298h50m25.363227917s) [noise=true , address=145.79.227.46:4605 , piece_id=OG2HB2OCB3X3X3BNCFIETDRZ5QCRCUDT5PVHOHRXC4FZ7KQM3RRQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.767076354 (298h50m25.363847388s) [noise=true , address=79.127.219.41:4928 , piece_id=WU43AECRGBCBAIX6TXV6PCWNHDYIP7E3ZS2MAG7WWWRLFFS4BHIA, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.767351880 (298h50m25.364122913s) [noise=true , address=77.90.31.21:4877 , piece_id=YSUHAWP4SJUX2SUNUOG6ONPLD7A3MDDYRZC4WIQUWMM4SMU7TPOQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.767738901 (298h50m25.364509934s) [noise=true , address=77.90.31.13:4629 , piece_id=EAXRUP4K2QSFUMA3TR4JEUTEC7OPLAWBCFRTX2JMR6G5GC54NXZQ, seg_pos={0 0}, upload=334705]: Uploading piece... Mar 18 21:50:27.864307706 (298h50m25.461078719s) [upload=334705]: Next returned nil segment Mar 18 21:50:27.864322635 (298h50m25.461093665s) [upload=334705]: Waiting for error group managing segments... Mar 18 21:50:27.870377313 (298h50m25.467148337s) [noise=true , address=77.90.31.21:4877 , piece_id=YSUHAWP4SJUX2SUNUOG6ONPLD7A3MDDYRZC4WIQUWMM4SMU7TPOQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.872155815 (298h50m25.468926846s) [noise=true , address=45.91.116.7:4009 , piece_id=L52PA4RFKPQX4IIDFY2C3QDAPSRURKN5YQQCTKEPYBHBIBSOMX3Q, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.872969727 (298h50m25.46974076s) [noise=true , address=77.90.31.13:4629 , piece_id=EAXRUP4K2QSFUMA3TR4JEUTEC7OPLAWBCFRTX2JMR6G5GC54NXZQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.873610155 (298h50m25.470381183s) [noise=true , address=145.79.227.3:4153 , piece_id=3T3DCDSKUUM3DSZHGP5KSC3FOY2NLYDPELOUPFUCWYWMAPOFP3UQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.874367621 (298h50m25.471138645s) [noise=true , address=79.127.219.41:4928 , piece_id=WU43AECRGBCBAIX6TXV6PCWNHDYIP7E3ZS2MAG7WWWRLFFS4BHIA, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.875119106 (298h50m25.471890156s) [noise=true , address=145.79.227.46:4605 , piece_id=OG2HB2OCB3X3X3BNCFIETDRZ5QCRCUDT5PVHOHRXC4FZ7KQM3RRQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.875841333 (298h50m25.472612362s) [noise=true , address=145.79.227.19:4081 , piece_id=6CCCUIQ7P67YLXPLTLLRKEZFW644G5KARTOILJGZ65VBFGNLNKTQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.877173575 (298h50m25.473944598s) [noise=true , address=145.79.227.42:4124 , piece_id=KPLOTO2MU3TCFEWQGBLVC2FXGFHNYZG5Y4AJGPGI2Z6YUSDAWZ5A, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.877546167 (298h50m25.474317193s) [noise=true , address=145.79.227.2:4113 , piece_id=LDADQAB7RHDTGQNCGVXUFWC4L5552LGT75M6F2BT763SO22H7VMQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.877649587 (298h50m25.474420614s) [noise=true , address=77.90.31.15:4697 , piece_id=4HIKVKB3PRE6VQ74UDXU7EKHIVJ3RMPJ457HHKB7RLFCLU223OZQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.878348093 (298h50m25.475119117s) [noise=true , address=145.79.227.27:4694 , piece_id=4EIWTOHMJJ6Z23OZBHLCG2NBNL55UR4PS5JDYB3SWAGFCZRPWSYA, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.878381649 (298h50m25.475152676s) [noise=true , address=145.79.227.31:4798 , piece_id=QDFKAJBFBBISOMGW5OEAOJFDVUALWPK46F33RZWVB7DGT2F2HMEA, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.878823621 (298h50m25.475594652s) [noise=true , address=145.79.227.16:4913 , piece_id=OQY2XYXGFI3WSELTXNDHTYXB57BJF2BDMPKOL2JUXUPC6N66A7NQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: <nil> Mar 18 21:50:27.878841757 (298h50m25.475612787s) [seg_pos={0 0}, upload=334705]: Satisfied cohort requirements after 13 pieces Mar 18 21:50:27.879139017 (298h50m25.47591017s) [noise=true , address=145.79.227.39:4019 , piece_id=TWN5U4ZM5WUFFDFXPY5PUEAEIWCC5LIKJNJBMELFHMYYL3DC3CPA, seg_pos={0 0}, upload=334705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ztmvXKmY6hicHZD7iejpcJCKFEhLwemtaYChAp8Po85TRQvGo): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:27.879152476 (298h50m25.475923508s) [noise=true , address=145.79.227.39:4019 , piece_id=TWN5U4ZM5WUFFDFXPY5PUEAEIWCC5LIKJNJBMELFHMYYL3DC3CPA, seg_pos={0 0}, upload=334705]: Long tail cancellation due to optimal threshold Mar 18 21:50:27.879216046 (298h50m25.475987079s) [noise=true , address=77.90.31.17:4774 , piece_id=EEMBISWBKYNHPN6XNTQNLHF5E74T4JMORFFFOWR7VQMGSYQEHG7Q, seg_pos={0 0}, upload=334705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:27.879228353 (298h50m25.475999388s) [noise=true , address=77.90.31.17:4774 , piece_id=EEMBISWBKYNHPN6XNTQNLHF5E74T4JMORFFFOWR7VQMGSYQEHG7Q, seg_pos={0 0}, upload=334705]: Long tail cancellation due to optimal threshold Mar 18 21:50:27.879469683 (298h50m25.476240742s) [noise=true , address=145.79.227.40:4055 , piece_id=LXUZUGBSW6W6ZFRBMB743PWQQSMBXOENK77NZIXGRQZYVU3UPWIQ, seg_pos={0 0}, upload=334705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SAWKJT2JEb9ReJUGDZ545Rp7LeQoCJKYsQgmrDA9csufPf8us): protocol: expected piece hash; context canceled; EOF Mar 18 21:50:27.879481428 (298h50m25.476252461s) [noise=true , address=145.79.227.40:4055 , piece_id=LXUZUGBSW6W6ZFRBMB743PWQQSMBXOENK77NZIXGRQZYVU3UPWIQ, seg_pos={0 0}, upload=334705]: Long tail cancellation due to optimal threshold Mar 18 21:50:27.879504862 (298h50m25.476275886s) [seg_pos={0 0}, upload=334705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:50:27.879611950 (298h50m25.476382979s) [upload=334705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:50:27.879686210 (298h50m25.47645724s) [upload=334705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:50:41.457484328 (298h50m39.054255341s) [upload=334705]: Done uploading segments... Mar 18 21:50:41.457621783 (298h50m39.05439281s) [upload=334705]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.033923914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 41.201555732s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:50:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 41.201589026s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................................XXXXXX.....X] duration: 30.002782748s trace: https://jaeger.dev.storj.io/trace/1eb8df1fb540f54 FAIL (2026-03-18 21:50:11 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:49:41.396329036 (298h49m38.993100062s) [upload=334695]: Starting upload Mar 18 21:49:41.396404588 (298h49m38.99317562s) [upload=334695]: Done starting upload Mar 18 21:49:41.396535766 (298h49m38.99330678s) [upload=334695]: Uploading segments... Mar 18 21:49:41.396805814 (298h49m38.993576842s) [upload=334695]: Got next segment. Inline: false Mar 18 21:49:41.396867639 (298h49m38.99363867s) [upload=334695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:49:41.396874843 (298h49m38.993645873s) [upload=334695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:49:49.534461304 (298h49m47.131232337s) [seg_pos={0 0}, upload=334695]: Begin upload segment... Mar 18 21:49:49.534533531 (298h49m47.131304559s) [seg_pos={0 0}, upload=334695]: Done begin upload segment. Mar 18 21:49:49.534572303 (298h49m47.131343336s) [noise=true , address=145.79.227.8:4331 , piece_id=KV546X6U7QFZDEMDY66AMTINB6C26J67ZM2ARKC7RVXQ3R5X7WMQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.539327654 (298h49m47.136098685s) [noise=true , address=145.79.227.28:4725 , piece_id=XGCBE5MTHZ76RFDF6F6UW4YPQB5HHHHQWVQ3X6OURTQSCGBZYBPQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.540257623 (298h49m47.137028655s) [noise=true , address=145.79.227.31:4802 , piece_id=TNRREIMGVGAWDHLHNZTORHWEZSR7IBU3U2UOJHWU4H4FCWHGIWMQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.540961510 (298h49m47.13773254s) [noise=true , address=145.79.227.25:4619 , piece_id=A66R2DXP5BGGXYETV5NWEEKRRGYWE4Y3LPZHBZKIPOZM7M4DLMUQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.542516962 (298h49m47.139287996s) [noise=true , address=145.79.227.44:4198 , piece_id=DKDOQTVR5J2HQVLZPKJDN7KZ63PXPGNNTS7JBHN2744EZKP3BFLA, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.542765126 (298h49m47.13953616s) [noise=true , address=145.79.227.48:4305 , piece_id=BBUQITWVOG25K5T3IXDZP45LQBHMFHL66DZP7WVU4KV6PKVQFQSQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.544555477 (298h49m47.141326531s) [noise=true , address=145.79.227.21:4479 , piece_id=X3HJLAQRUX254D4TRP4PP6B4I2OC4JSHSDNFKIJSHIBX2YN6CMQA, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.544877491 (298h49m47.141648526s) [noise=true , address=145.79.227.40:4051 , piece_id=QYQKASM7KXV5IN3BVOYRY3DFAZCI5X7MR2UIHELJXAD2VV6HYI4Q, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.546429010 (298h49m47.143200037s) [noise=true , address=145.79.227.33:4870 , piece_id=4L2NPSYMKBNPHN4PKOFVRA5ARQQEY724S4XYKN6QNY44YSDI5ILQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.547014407 (298h49m47.143785441s) [noise=true , address=145.79.227.26:4656 , piece_id=NXNOO6YGVKK6FZUBPVOV3LTZP2G35S3C64JZBXU4WJF5GVO3VRRA, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.548667575 (298h49m47.145438609s) [noise=true , address=145.79.227.47:4268 , piece_id=4TFTJQKY4NVRHQV7ULC24ZYHGWBAW7ACHAF2A4ZERK2CUQ6PCK3Q, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.549046069 (298h49m47.145817104s) [noise=true , address=145.79.227.34:4908 , piece_id=DJ54GRSB4RD6GJYMWL3WK4K7BFZHKAGV5KCOKETBBBQ5JODQSV4Q, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.550964709 (298h49m47.147735742s) [noise=true , address=45.91.116.5:4298 , piece_id=3S73KYWL2APWV5KX4G5G475OFFQHYHCD3UDAIUTU5KGKV5Y6SIKQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.551074317 (298h49m47.147845352s) [noise=true , address=145.79.227.22:4514 , piece_id=WUEY2SB3BXECJKP7JV3X6DY2OSP3EEA5YZFVQ4DYZDH5AWHWO6ZQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.553710512 (298h49m47.150481545s) [noise=true , address=145.79.227.42:4122 , piece_id=LM45OJBJ3VEZW2NB3NMQBPWCUP2Z3BZWA444F46RF5BYWCSUWX2A, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.555903844 (298h49m47.152674873s) [noise=true , address=145.79.227.50:4377 , piece_id=HN6NI2XFA57JDTWII4QMURCHVID4IZWGA26IBYKVX5CMTTCWN6SQ, seg_pos={0 0}, upload=334695]: Uploading piece... Mar 18 21:49:49.625872610 (298h49m47.222643645s) [upload=334695]: Next returned nil segment Mar 18 21:49:49.625889687 (298h49m47.22266072s) [upload=334695]: Waiting for error group managing segments... Mar 18 21:49:49.631517326 (298h49m47.228288356s) [noise=true , address=45.91.116.5:4298 , piece_id=3S73KYWL2APWV5KX4G5G475OFFQHYHCD3UDAIUTU5KGKV5Y6SIKQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.634110813 (298h49m47.23088184s) [noise=true , address=145.79.227.8:4331 , piece_id=KV546X6U7QFZDEMDY66AMTINB6C26J67ZM2ARKC7RVXQ3R5X7WMQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.634685493 (298h49m47.231456519s) [noise=true , address=145.79.227.28:4725 , piece_id=XGCBE5MTHZ76RFDF6F6UW4YPQB5HHHHQWVQ3X6OURTQSCGBZYBPQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.635104106 (298h49m47.231875134s) [noise=true , address=145.79.227.25:4619 , piece_id=A66R2DXP5BGGXYETV5NWEEKRRGYWE4Y3LPZHBZKIPOZM7M4DLMUQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.636626821 (298h49m47.233397903s) [noise=true , address=145.79.227.26:4656 , piece_id=NXNOO6YGVKK6FZUBPVOV3LTZP2G35S3C64JZBXU4WJF5GVO3VRRA, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.637376563 (298h49m47.234147588s) [noise=true , address=145.79.227.42:4122 , piece_id=LM45OJBJ3VEZW2NB3NMQBPWCUP2Z3BZWA444F46RF5BYWCSUWX2A, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.637922767 (298h49m47.234693795s) [noise=true , address=145.79.227.50:4377 , piece_id=HN6NI2XFA57JDTWII4QMURCHVID4IZWGA26IBYKVX5CMTTCWN6SQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.638689661 (298h49m47.235460692s) [noise=true , address=145.79.227.22:4514 , piece_id=WUEY2SB3BXECJKP7JV3X6DY2OSP3EEA5YZFVQ4DYZDH5AWHWO6ZQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.639525181 (298h49m47.236296209s) [noise=true , address=145.79.227.31:4802 , piece_id=TNRREIMGVGAWDHLHNZTORHWEZSR7IBU3U2UOJHWU4H4FCWHGIWMQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.640283830 (298h49m47.237054853s) [noise=true , address=145.79.227.34:4908 , piece_id=DJ54GRSB4RD6GJYMWL3WK4K7BFZHKAGV5KCOKETBBBQ5JODQSV4Q, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.641019298 (298h49m47.237790321s) [noise=true , address=145.79.227.33:4870 , piece_id=4L2NPSYMKBNPHN4PKOFVRA5ARQQEY724S4XYKN6QNY44YSDI5ILQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.641977613 (298h49m47.238748636s) [noise=true , address=145.79.227.48:4305 , piece_id=BBUQITWVOG25K5T3IXDZP45LQBHMFHL66DZP7WVU4KV6PKVQFQSQ, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.649203074 (298h49m47.24597411s) [noise=true , address=145.79.227.21:4479 , piece_id=X3HJLAQRUX254D4TRP4PP6B4I2OC4JSHSDNFKIJSHIBX2YN6CMQA, seg_pos={0 0}, upload=334695]: Done uploading piece. err: <nil> Mar 18 21:49:49.649231727 (298h49m47.246002754s) [seg_pos={0 0}, upload=334695]: Satisfied cohort requirements after 13 pieces Mar 18 21:49:49.649509932 (298h49m47.246280958s) [noise=true , address=145.79.227.40:4051 , piece_id=QYQKASM7KXV5IN3BVOYRY3DFAZCI5X7MR2UIHELJXAD2VV6HYI4Q, seg_pos={0 0}, upload=334695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:49.649519672 (298h49m47.246290696s) [noise=true , address=145.79.227.40:4051 , piece_id=QYQKASM7KXV5IN3BVOYRY3DFAZCI5X7MR2UIHELJXAD2VV6HYI4Q, seg_pos={0 0}, upload=334695]: Long tail cancellation due to optimal threshold Mar 18 21:49:49.649819781 (298h49m47.246590813s) [noise=true , address=145.79.227.44:4198 , piece_id=DKDOQTVR5J2HQVLZPKJDN7KZ63PXPGNNTS7JBHN2744EZKP3BFLA, seg_pos={0 0}, upload=334695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:49.649827084 (298h49m47.246598108s) [noise=true , address=145.79.227.44:4198 , piece_id=DKDOQTVR5J2HQVLZPKJDN7KZ63PXPGNNTS7JBHN2744EZKP3BFLA, seg_pos={0 0}, upload=334695]: Long tail cancellation due to optimal threshold Mar 18 21:49:49.650067331 (298h49m47.24683836s) [noise=true , address=145.79.227.47:4268 , piece_id=4TFTJQKY4NVRHQV7ULC24ZYHGWBAW7ACHAF2A4ZERK2CUQ6PCK3Q, seg_pos={0 0}, upload=334695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:49.650076469 (298h49m47.246847493s) [noise=true , address=145.79.227.47:4268 , piece_id=4TFTJQKY4NVRHQV7ULC24ZYHGWBAW7ACHAF2A4ZERK2CUQ6PCK3Q, seg_pos={0 0}, upload=334695]: Long tail cancellation due to optimal threshold Mar 18 21:49:49.650101485 (298h49m47.24687251s) [seg_pos={0 0}, upload=334695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:49:49.650198934 (298h49m47.246969958s) [upload=334695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:49:49.650206030 (298h49m47.246977055s) [upload=334695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:49:58.120518059 (298h49m55.717289082s) [upload=334695]: Done uploading segments... Mar 18 21:49:58.120540106 (298h49m55.717311129s) [upload=334695]: Upload finished. err: <nil> Duration: 30.002829309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..........................................X......XXXXXX.X...] duration: 21.274221572s trace: https://jaeger.dev.storj.io/trace/21db7836809bef5e FAIL (2026-03-18 21:50:10 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:40.920904207 (298h47m38.51767523s) [upload=334662]: Starting upload Mar 18 21:47:40.920983209 (298h47m38.517754224s) [upload=334662]: Done starting upload Mar 18 21:47:40.921096404 (298h47m38.517867417s) [upload=334662]: Uploading segments... Mar 18 21:47:40.921141118 (298h47m38.517912146s) [upload=334662]: Got next segment. Inline: true Mar 18 21:47:40.921158709 (298h47m38.51792974s) [upload=334662]: Waiting for error group managing segments... Mar 18 21:47:40.921211193 (298h47m38.517982206s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:40.921216807 (298h47m38.51798782s) [upload=334662]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:40.921236184 (298h47m38.518007211s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:57.835665861 (298h47m55.432436892s) [upload=334662]: Done uploading segments... Mar 18 21:47:57.835696771 (298h47m55.432467801s) [upload=334662]: Upload finished. err: <nil> Duration:* 21.274249527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXXX] duration: 5.001411016s trace: https://jaeger.dev.storj.io/trace/44719122ecfe539a FAIL (2026-03-18 21:49:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001443793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................................XX.....X.XXXXXX.X.] duration: 22.649593953s trace: https://jaeger.dev.storj.io/trace/2fb1108aa58b9f75 FAIL (2026-03-18 21:49:58 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.649619156s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [........................................X....X...XXX..XX.X..] duration: 29.046992508s trace: https://jaeger.dev.storj.io/trace/54d45aef5a0b8ced FAIL (2026-03-18 21:49:53 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 29.047092446s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 33.31195686s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:49:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 33.311984698s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XXXXXX.....] duration: 21.057376806s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:49:34 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 21.057410782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...........................................X......XXXXXX.X..] duration: 25.030373828s trace: https://jaeger.dev.storj.io/trace/21db7836809bef5e FAIL (2026-03-18 21:49:23 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:40.920904207 (298h47m38.51767523s) [upload=334662]: Starting upload Mar 18 21:47:40.920983209 (298h47m38.517754224s) [upload=334662]: Done starting upload Mar 18 21:47:40.921096404 (298h47m38.517867417s) [upload=334662]: Uploading segments... Mar 18 21:47:40.921141118 (298h47m38.517912146s) [upload=334662]: Got next segment. Inline: true Mar 18 21:47:40.921158709 (298h47m38.51792974s) [upload=334662]: Waiting for error group managing segments... Mar 18 21:47:40.921211193 (298h47m38.517982206s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:40.921216807 (298h47m38.51798782s) [upload=334662]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:40.921236184 (298h47m38.518007211s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:57.835665861 (298h47m55.432436892s) [upload=334662]: Done uploading segments... Mar 18 21:47:57.835696771 (298h47m55.432467801s) [upload=334662]: Upload finished. err: <nil> Duration:* 25.030480792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........................................XX.....X.XXXXXX.X] duration: 30.006286124s trace: https://jaeger.dev.storj.io/trace/2fb1108aa58b9f75 FAIL (2026-03-18 21:49:04 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006314588s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXXX] duration: 5.001617171s trace: https://jaeger.dev.storj.io/trace/7a81cfd658547b84 FAIL (2026-03-18 21:49:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001650687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XXXXXX....] duration: 21.460693346s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:49:00 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 21.460716628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 33.0363625s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:48:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 33.03639764s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [.........................................X....X...XXX..XX.X.] duration: 21.250169843s trace: https://jaeger.dev.storj.io/trace/54d45aef5a0b8ced FAIL (2026-03-18 21:48:45 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 21.250202134s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXXX] duration: 5.001618882s trace: https://jaeger.dev.storj.io/trace/6a11cded55c3689b FAIL (2026-03-18 21:48:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001647668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XXXXXX...] duration: 21.252716265s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:48:38 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 21.252814964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [............................................X......XXXXXX.X.] duration: 21.130061836s trace: https://jaeger.dev.storj.io/trace/21db7836809bef5e FAIL (2026-03-18 21:48:32 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:40.920904207 (298h47m38.51767523s) [upload=334662]: Starting upload Mar 18 21:47:40.920983209 (298h47m38.517754224s) [upload=334662]: Done starting upload Mar 18 21:47:40.921096404 (298h47m38.517867417s) [upload=334662]: Uploading segments... Mar 18 21:47:40.921141118 (298h47m38.517912146s) [upload=334662]: Got next segment. Inline: true Mar 18 21:47:40.921158709 (298h47m38.51792974s) [upload=334662]: Waiting for error group managing segments... Mar 18 21:47:40.921211193 (298h47m38.517982206s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:40.921216807 (298h47m38.51798782s) [upload=334662]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:40.921236184 (298h47m38.518007211s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:57.835665861 (298h47m55.432436892s) [upload=334662]: Done uploading segments... Mar 18 21:47:57.835696771 (298h47m55.432467801s) [upload=334662]: Upload finished. err: <nil> Duration:* 21.13008745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [.............................................X......XXXXXX.X] duration: 30.001484885s trace: https://jaeger.dev.storj.io/trace/21db7836809bef5e FAIL (2026-03-18 21:48:10 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:40.920904207 (298h47m38.51767523s) [upload=334662]: Starting upload Mar 18 21:47:40.920983209 (298h47m38.517754224s) [upload=334662]: Done starting upload Mar 18 21:47:40.921096404 (298h47m38.517867417s) [upload=334662]: Uploading segments... Mar 18 21:47:40.921141118 (298h47m38.517912146s) [upload=334662]: Got next segment. Inline: true Mar 18 21:47:40.921158709 (298h47m38.51792974s) [upload=334662]: Waiting for error group managing segments... Mar 18 21:47:40.921211193 (298h47m38.517982206s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:40.921216807 (298h47m38.51798782s) [upload=334662]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:40.921236184 (298h47m38.518007211s) [upload=334662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:57.835665861 (298h47m55.432436892s) [upload=334662]: Done uploading segments... Mar 18 21:47:57.835696771 (298h47m55.432467801s) [upload=334662]: Upload finished. err: <nil> Duration:* 30.00151224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XXXXXX..] duration: 21.66015355s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:48:03 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 21.660184257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..........................................X....X...XXX..XX.X] duration: 30.000817054s trace: https://jaeger.dev.storj.io/trace/54d45aef5a0b8ced FAIL (2026-03-18 21:47:58 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000845203s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............................................XX.....X.XXXXXX.] duration: 26.531071482s trace: https://jaeger.dev.storj.io/trace/3c9a1e00c2f98236 FAIL (2026-03-18 21:47:52 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.531173291s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 36.872795018s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:47:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 36.872816165s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXXX] duration: 5.000674956s trace: https://jaeger.dev.storj.io/trace/62ed484aab1d6fa FAIL (2026-03-18 21:47:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000694992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..............................................X......XXXXXX.] duration: 21.300994777s trace: https://jaeger.dev.storj.io/trace/686f2d7df45e94a0 FAIL (2026-03-18 21:47:29 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:09.173063667 (298h46m6.769834694s) [upload=334632]: Starting upload Mar 18 21:46:09.173177671 (298h46m6.769948688s) [upload=334632]: Done starting upload Mar 18 21:46:09.173313386 (298h46m6.770084405s) [upload=334632]: Uploading segments... Mar 18 21:46:09.173432086 (298h46m6.770203119s) [upload=334632]: Got next segment. Inline: true Mar 18 21:46:09.173440487 (298h46m6.770211518s) [upload=334632]: Waiting for error group managing segments... Mar 18 21:46:09.173519213 (298h46m6.770290236s) [upload=334632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:09.173539379 (298h46m6.770310402s) [upload=334632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:46:09.173543407 (298h46m6.770314431s) [upload=334632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:39.138331939 (298h46m36.735102972s) [upload=334632]: Done uploading segments... Mar 18 21:46:39.138402060 (298h46m36.735173092s) [upload=334632]: Upload finished. err: metaclient: context deadline exceeded Duration:* 21.301028974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 57.435088331s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:47:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 57.435122628s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 25.210750163s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:46:58 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 25.21078136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X....X...XXX..XX.] duration: 28.958101139s trace: https://jaeger.dev.storj.io/trace/4c63b81fd9142876 FAIL (2026-03-18 21:46:58 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 28.958130376s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXXX] duration: 5.001369712s trace: https://jaeger.dev.storj.io/trace/7a972995ff01e0a FAIL (2026-03-18 21:46:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001401371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XX.....X.XXXXXX] duration: 30.006285976s trace: https://jaeger.dev.storj.io/trace/3c9a1e00c2f98236 FAIL (2026-03-18 21:46:42 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.006324916s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X......XXXXXX] duration: 30.001488054s trace: https://jaeger.dev.storj.io/trace/686f2d7df45e94a0 FAIL (2026-03-18 21:46:39 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:09.173063667 (298h46m6.769834694s) [upload=334632]: Starting upload Mar 18 21:46:09.173177671 (298h46m6.769948688s) [upload=334632]: Done starting upload Mar 18 21:46:09.173313386 (298h46m6.770084405s) [upload=334632]: Uploading segments... Mar 18 21:46:09.173432086 (298h46m6.770203119s) [upload=334632]: Got next segment. Inline: true Mar 18 21:46:09.173440487 (298h46m6.770211518s) [upload=334632]: Waiting for error group managing segments... Mar 18 21:46:09.173519213 (298h46m6.770290236s) [upload=334632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:09.173539379 (298h46m6.770310402s) [upload=334632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:46:09.173543407 (298h46m6.770314431s) [upload=334632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:39.138331939 (298h46m36.735102972s) [upload=334632]: Done uploading segments... Mar 18 21:46:39.138402060 (298h46m36.735173092s) [upload=334632]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001511804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.....XXXXXXXXXXXXXXXX..XXXXXXXXX] duration: 5.001396818s trace: https://jaeger.dev.storj.io/trace/5bc5f02a2d12d1d8 FAIL (2026-03-18 21:46:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001476234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.002313112s trace: https://jaeger.dev.storj.io/trace/44e2f6a9b7e0b1c5 FAIL (2026-03-18 21:46:15 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:45:45.371880476 (298h45m42.968651502s) [upload=334628]: Starting upload Mar 18 21:45:45.371981279 (298h45m42.968752303s) [upload=334628]: Done starting upload Mar 18 21:45:45.372033719 (298h45m42.968804734s) [upload=334628]: Uploading segments... Mar 18 21:45:45.372253721 (298h45m42.969024742s) [upload=334628]: Got next segment. Inline: false Mar 18 21:45:45.372296945 (298h45m42.969067968s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:45:45.372318118 (298h45m42.96908914s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:45:57.578609786 (298h45m55.17538081s) [seg_pos={0 0}, upload=334628]: Begin upload segment... Mar 18 21:45:57.578679804 (298h45m55.175450835s) [seg_pos={0 0}, upload=334628]: Done begin upload segment. Mar 18 21:45:57.578738604 (298h45m55.175509628s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579192283 (298h45m55.175963318s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.579757590 (298h45m55.176528619s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580174439 (298h45m55.176945474s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580300533 (298h45m55.177071559s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580737580 (298h45m55.177508615s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.580828687 (298h45m55.17759971s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581254886 (298h45m55.17802592s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581287802 (298h45m55.178058825s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581798418 (298h45m55.178569451s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.581805571 (298h45m55.178576612s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582254708 (298h45m55.179025742s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582406043 (298h45m55.17917707s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582685555 (298h45m55.179456583s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.582929046 (298h45m55.179700082s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.583538331 (298h45m55.180309362s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Uploading piece... Mar 18 21:45:57.668491005 (298h45m55.265262036s) [upload=334628]: Next returned nil segment Mar 18 21:45:57.668508856 (298h45m55.265279889s) [upload=334628]: Waiting for error group managing segments... Mar 18 21:45:57.680368043 (298h45m55.277139073s) [noise=true , address=145.79.227.10:4405 , piece_id=WR43VEBW352SJOSKSL32NCNHYBRWAKNFJGU6XIF6KDTYAVCIRI2Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.680795091 (298h45m55.277566115s) [noise=true , address=145.79.227.6:4263 , piece_id=LGBSGTW5BHKTO7GEPU5JJXVFNGP5BGWD4SI5Y6P7NK75E5S7DEUA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681371676 (298h45m55.278142699s) [noise=true , address=145.79.227.7:4295 , piece_id=NFQ6ZVJHR67D2K4KHLCUZNMO3GEH4TNJH4CZETQX27V3JJMF7JHA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.681772316 (298h45m55.278543339s) [noise=true , address=145.79.227.17:4975 , piece_id=DCCOGJC4WDANCYHOCIWMCOKS33WCM3YAG5PED6CQYGC4GRTGWMSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682139829 (298h45m55.278910852s) [noise=true , address=77.90.31.10:4519 , piece_id=QAVJH6L4NDLZM2BX3C7OYR4BCJEGXWOFJOBFFFLDHNMGXYUT3PJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.682569175 (298h45m55.279340203s) [noise=true , address=77.90.31.18:4805 , piece_id=UPA6BGG75ZWTB3W55R3XANNZP4GGYW6D5NPUSCVYD4IOCRSVENOA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683392374 (298h45m55.280163397s) [noise=true , address=145.79.227.38:4980 , piece_id=U2MZGZUPAB5YN4O2AWG6UB7JTYDZ45LS3WXHXT4EUXB36ZHLAE4Q, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683592532 (298h45m55.280363544s) [noise=true , address=45.91.116.6:4989 , piece_id=EIN27PK6P4UVKBNP4VMVHR4DTXIVGBNRTOWXQ3FS3CY6KRXI6SLA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.683873521 (298h45m55.280644543s) [noise=true , address=145.79.227.14:4839 , piece_id=WEEVUJAR5PNXTQS6Y5Q5TK7Q7KPKAXPP3QRI4SYHGZHH2WGWYI2A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684245051 (298h45m55.281016078s) [noise=true , address=145.79.227.40:4052 , piece_id=QI4IQFCIAFTWE6B5OACHH2HERXPB3LMYDTTQMWN7XMUY3HC5W56A, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684547385 (298h45m55.281318408s) [noise=true , address=145.79.227.12:4771 , piece_id=LCA3OL7XVNFZ3WLEWSJF2JUEPWTLBHNZOSZVCMKYK2EXIMSRJRBQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.684690723 (298h45m55.281461751s) [noise=true , address=145.79.227.8:4331 , piece_id=2HN4MKZN5ABZLAVBDMV4IU5WMF62G45MTTD5LYSIFRQZ2MUL3VCA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685141120 (298h45m55.281912133s) [noise=true , address=145.79.227.9:4369 , piece_id=JO23YSAZDKV244QF264CUWOKJXB44X47OBERZ6Y5HUADDU6YPJYQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: <nil> Mar 18 21:45:57.685153222 (298h45m55.281924235s) [seg_pos={0 0}, upload=334628]: Satisfied cohort requirements after 13 pieces Mar 18 21:45:57.685356456 (298h45m55.282127485s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685364416 (298h45m55.282135441s) [noise=true , address=145.79.227.5:4224 , piece_id=KIVWOR5JWMUVQF3KZJAUE4BMYRLYJH6OYAXV3SVTOSYP6JWBMLSQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685670388 (298h45m55.282441414s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685680835 (298h45m55.282451868s) [noise=true , address=145.79.227.44:4193 , piece_id=BHWJLFVYYPUYRWQV5GFM24HI5FMSKUGRT2FRZWLRC35MIVQXVWTA, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685708964 (298h45m55.282479996s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Mar 18 21:45:57.685719246 (298h45m55.282490272s) [noise=true , address=145.79.227.34:4909 , piece_id=KWC2BCCADUPD6TNHIXCXS5E4EVFIHUZXIBGNHESUQCHKV4J2MXJQ, seg_pos={0 0}, upload=334628]: Long tail cancellation due to optimal threshold Mar 18 21:45:57.685740822 (298h45m55.282511845s) [seg_pos={0 0}, upload=334628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:45:57.685818688 (298h45m55.282589711s) [upload=334628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:45:57.685824343 (298h45m55.282595366s) [upload=334628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:06.163960625 (298h46m3.760731642s) [upload=334628]: Done uploading segments... Mar 18 21:46:06.163982123 (298h46m3.760753136s) [upload=334628]: Upload finished. err: <nil> Duration: 30.002414972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 37.076367293s trace: https://jaeger.dev.storj.io/trace/2ab94940f3da3187 FAIL (2026-03-18 21:46:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:41.272084084 (298h41m38.868855112s) [upload=334559]: Starting upload Mar 18 21:41:41.272169036 (298h41m38.868940066s) [upload=334559]: Done starting upload Mar 18 21:41:41.272251155 (298h41m38.869022168s) [upload=334559]: Uploading segments... Mar 18 21:41:41.272358183 (298h41m38.869129211s) [upload=334559]: Got next segment. Inline: true Mar 18 21:41:41.272367512 (298h41m38.869138542s) [upload=334559]: Waiting for error group managing segments... Mar 18 21:41:41.272525860 (298h41m38.869296906s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:41.272550748 (298h41m38.869321776s) [upload=334559]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:41.272555523 (298h41m38.86932655s) [upload=334559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:30.253452076 (298h42m27.850223099s) [upload=334559]: Done uploading segments... Mar 18 21:42:30.253475139 (298h42m27.850246163s) [upload=334559]: Upload finished. err: <nil> Duration:* 37.076395611s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX.....X.XXXXX] duration: 30.005636082s trace: https://jaeger.dev.storj.io/trace/751a160f67cb679f FAIL (2026-03-18 21:46:04 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005719717s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [............................................X....X...XXX..XX] duration: 30.001699749s trace: https://jaeger.dev.storj.io/trace/4c63b81fd9142876 FAIL (2026-03-18 21:45:56 UTC) operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001802307s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)