Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 164545 reports (Page 127 of 3291)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration:
418.213896ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:45:53 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 418.260406ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........]
duration:
496.554822ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:45:29 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 496.619457ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration:
509.950466ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:44:34 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 510.019057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration:
471.903647ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:43:53 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 472.065664ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration:
438.290797ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:42:42 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 438.317548ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration:
1.808322396s
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:42:07 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 1.808347651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...]
duration:
463.833177ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:41:19 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 463.857721ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration:
519.288473ms
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:40:32 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 519.316867ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration:
2.297334628s
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:39:57 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.297350132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X]
duration:
30.027229237s
trace:
https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd
FAIL (2025-12-09 12:39:54 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload
Dec  9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload
Dec  9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments...
Dec  9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false
Dec  9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment...
Dec  9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment.
Dec  9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece...
Dec  9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698     , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590     , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec  9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment
Dec  9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments...
Dec  9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807     , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367    , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942   , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086   , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122   , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582   , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157   , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834   , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797   , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473   , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553     , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878     , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil>
Dec  9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842     , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734     , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments...
Dec  9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.027256929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................]
duration:
449.216498ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:33:03 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 449.301102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration:
494.554209ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:32:21 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 494.61508ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration:
480.129474ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:31:57 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 480.212886ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................]
duration:
399.124584ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:30:50 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 399.168707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration:
429.161382ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:30:30 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 429.222415ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
480.81609ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:29:41 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 480.841095ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
443.941597ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:28:59 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 443.965816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration:
439.932664ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:27:48 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 439.969055ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................]
duration:
514.615822ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:27:04 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 514.665569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
571.479999ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:26:04 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 571.508071ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration:
500.287386ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:25:00 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 500.334783ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................]
duration:
425.089343ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:23:52 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 425.113402ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................]
duration:
470.198796ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:22:56 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 470.227775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration:
820.839613ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:22:17 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 820.866544ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration:
381.796505ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:21:07 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 381.848972ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................]
duration:
427.194015ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:20:07 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 427.229992ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................]
duration:
388.315378ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:19:23 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 388.339778ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration:
428.414188ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:19:07 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 428.444092ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration:
527.174232ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:18:09 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 527.210269ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................]
duration:
468.107881ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:16:59 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 468.131456ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration:
394.900911ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:16:40 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 394.923637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration:
376.278737ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:16:14 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 376.306585ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration:
432.087439ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:15:28 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 432.106512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration:
372.056202ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:14:34 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 372.098383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................]
duration:
466.281778ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:13:36 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 466.341665ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration:
439.778147ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:12:55 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 439.81754ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration:
443.420761ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:12:29 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 443.444225ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................]
duration:
376.534399ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:11:47 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 376.565435ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................]
duration:
434.517631ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:10:58 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 434.545449ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................]
duration:
408.742994ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:09:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 408.76226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................]
duration:
590.551272ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:09:02 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 590.584993ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................]
duration:
478.435297ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:08:44 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 478.462784ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................]
duration:
433.521519ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:08:17 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 433.553673ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................]
duration:
415.14352ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:07:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 415.194162ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................]
duration:
359.94263ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:06:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 359.967427ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................]
duration:
505.96333ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:05:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 506.077776ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................]
duration:
479.246361ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:05:02 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 479.310529ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................]
duration:
403.959664ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:04:34 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 403.985463ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................]
duration:
505.645585ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:03:22 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 505.672654ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................]
duration:
493.822504ms
trace:
https://jaeger.dev.storj.io/trace/305bb3403e570649
FAIL (2025-12-09 12:02:28 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec  9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload
Dec  9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload
Dec  9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments...
Dec  9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false
Dec  9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec  9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec  9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment...
Dec  9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment.
Dec  9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece...
Dec  9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627     , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841     , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec  9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment
Dec  9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments...
Dec  9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869   , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583   , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655   , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581   , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836   , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798   , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840   , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086   , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545   , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911   , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119    , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194   , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil>
Dec  9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698     , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735     , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF
Dec  9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec  9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments...
Dec  9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 493.907054ms core uplink us-select-1