Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 164605 reports (Page 127 of 3293)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........]
duration:
453.463535ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:08:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 453.581506ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........]
duration:
470.893629ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:07:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.930145ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......]
duration:
430.796055ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:07:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 430.847044ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......]
duration:
464.801191ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:06:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 464.815893ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....]
duration:
482.526438ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:06:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 483.103484ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....]
duration:
475.465858ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:05:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.492203ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...]
duration:
497.619581ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:05:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.883971ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..]
duration:
562.794808ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:04:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.822626ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.]
duration:
623.342178ms
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:03:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 623.361468ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X]
duration:
5.00131386s
trace:
https://jaeger.dev.storj.io/trace/4718e63560644445
FAIL (2025-12-09 16:02:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001336113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................]
duration:
398.93788ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:26:00 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 398.966166ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration:
521.808484ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:25:03 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 521.83819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration:
454.699087ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:23:55 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 454.734983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................]
duration:
505.888926ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:23:09 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 505.905413ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration:
560.794869ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:22:16 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 560.864005ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
489.445058ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:21:08 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 489.471976ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
494.195921ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:20:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 494.218798ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration:
471.08053ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:20:09 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 471.109362ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................]
duration:
574.857799ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:19:44 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 574.881042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
478.296898ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:18:48 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 478.355625ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration:
457.503057ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:17:37 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 457.614781ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................]
duration:
457.842025ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:16:27 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 457.864909ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................]
duration:
561.929844ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:16:06 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 561.966432ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration:
481.796475ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:15:03 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 481.827811ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration:
470.883922ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:14:02 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 471.232181ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................]
duration:
404.393913ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:13:35 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 404.489695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................]
duration:
425.718724ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:12:38 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 425.786518ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration:
479.024238ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:11:27 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 479.065045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration:
455.562839ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:10:49 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 455.590406ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................]
duration:
455.260245ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:09:38 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 455.286881ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration:
568.365386ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:08:33 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 568.450763ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration:
499.870873ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:08:15 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 499.905781ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration:
455.848074ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:07:35 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 455.979949ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration:
475.075661ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:06:32 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 475.150882ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................]
duration:
509.280459ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:06:00 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 509.310247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration:
460.261272ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:04:51 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 460.290873ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration:
521.019513ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:03:52 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 521.051231ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................]
duration:
514.240455ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:03:08 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 514.284017ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................]
duration:
503.66699ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:01:59 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 503.698383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................]
duration:
449.141063ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:01:17 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 449.172974ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................]
duration:
427.509728ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:00:44 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 427.538816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................]
duration:
501.873127ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 13:00:21 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 501.929193ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................]
duration:
527.076382ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:59:51 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 527.106144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................]
duration:
465.776124ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:59:18 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 465.818614ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................]
duration:
400.848758ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:58:24 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 400.875085ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................]
duration:
455.940537ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:57:43 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 455.966924ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................]
duration:
566.386383ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:57:03 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 566.433168ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................]
duration:
535.180569ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:56:16 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 535.263046ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................]
duration:
552.292185ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:55:08 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 552.315906ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................]
duration:
417.075593ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:54:14 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 417.106063ms core uplink us-select-1