Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 418.213896ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:45:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.260406ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 496.554822ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:45:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.619457ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 509.950466ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.019057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 471.903647ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.065664ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 438.290797ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:42:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.317548ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.808322396s trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:42:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.808347651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 463.833177ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:41:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.857721ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 519.288473ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:40:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.316867ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.297334628s trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:39:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.297350132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.027229237s trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:39:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027256929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 449.216498ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:33:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.301102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 494.554209ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.61508ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 480.129474ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.212886ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 399.124584ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:30:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.168707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 429.161382ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:30:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.222415ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 480.81609ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:29:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.841095ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 443.941597ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:28:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.965816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 439.932664ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:27:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.969055ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 514.615822ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.665569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 571.479999ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:26:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.508071ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 500.287386ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:25:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 500.334783ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 425.089343ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.113402ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 470.198796ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:22:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.227775ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 820.839613ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:22:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 820.866544ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 381.796505ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:21:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.848972ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 427.194015ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:20:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.229992ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 388.315378ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.339778ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 428.414188ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:19:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.444092ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 527.174232ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:18:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 527.210269ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 468.107881ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 468.131456ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 394.900911ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:16:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.923637ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 376.278737ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:16:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.306585ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 432.087439ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.106512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 372.056202ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 372.098383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 466.281778ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:13:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.341665ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 439.778147ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:12:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.81754ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 443.420761ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 443.444225ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 376.534399ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.565435ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 434.517631ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.545449ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 408.742994ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 408.76226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 590.551272ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 590.584993ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 478.435297ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:08:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.462784ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 433.521519ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:08:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.553673ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 415.14352ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:07:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.194162ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 359.94263ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:06:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 359.967427ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 505.96333ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:05:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.077776ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 479.246361ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:05:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.310529ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 403.959664ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.985463ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 505.645585ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.672654ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 493.822504ms trace: https://jaeger.dev.storj.io/trace/305bb3403e570649 FAIL (2025-12-09 12:02:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 11:48:49.157358580 (92h20m49.985066971s) [upload=95998]: Starting upload Dec 9 11:48:49.157475603 (92h20m49.985184101s) [upload=95998]: Done starting upload Dec 9 11:48:49.157569415 (92h20m49.985277796s) [upload=95998]: Uploading segments... Dec 9 11:48:49.157804983 (92h20m49.985513375s) [upload=95998]: Got next segment. Inline: false Dec 9 11:48:49.157855755 (92h20m49.985564146s) [upload=95998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:48:49.157861404 (92h20m49.985569796s) [upload=95998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:48:49.183100768 (92h20m50.010809165s) [seg_pos={0 0}, upload=95998]: Begin upload segment... Dec 9 11:48:49.183176342 (92h20m50.010884747s) [seg_pos={0 0}, upload=95998]: Done begin upload segment. Dec 9 11:48:49.183238885 (92h20m50.010947282s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183281606 (92h20m50.010990006s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.183978238 (92h20m50.011686622s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184478799 (92h20m50.012187181s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.184924558 (92h20m50.012632939s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185399543 (92h20m50.013107943s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.185868871 (92h20m50.013577266s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186227516 (92h20m50.013935914s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.186578710 (92h20m50.01428711s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187207023 (92h20m50.014915416s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.187730444 (92h20m50.015438842s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188255898 (92h20m50.01596431s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.188697539 (92h20m50.016405931s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189240329 (92h20m50.016948711s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.189738149 (92h20m50.01744655s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.190314906 (92h20m50.018023307s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Uploading piece... Dec 9 11:48:49.214710654 (92h20m50.042419054s) [noise=true , address=77.90.31.13:4627 , piece_id=PWJRAWUEEM4NV2RZNN4VAP76WPLODE3RAIXZAWJENE3RYOPVHRNQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.215276408 (92h20m50.042984805s) [noise=true , address=77.90.31.19:4841 , piece_id=4KW3OOCJKYS27OVCIUDCEPFF3VL673SUCWPXTASWHOGMBXJ7VHFQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 11:48:49.251869222 (92h20m50.079577614s) [upload=95998]: Next returned nil segment Dec 9 11:48:49.251884689 (92h20m50.07959308s) [upload=95998]: Waiting for error group managing segments... Dec 9 11:48:49.262818385 (92h20m50.090526776s) [noise=true , address=145.79.227.33:4869 , piece_id=BXRVUIAN6K52FJW2CB4ZYGAQH3ZPYRJT3WILPLG5IX55RMAUCHVA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.263885209 (92h20m50.091593604s) [noise=true , address=145.79.227.24:4583 , piece_id=IZKUUI4BAOVUS264GFQHYXRIBAV4NDST2N5BYHMJEPXBUZOGFPMQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264409508 (92h20m50.092117899s) [noise=true , address=145.79.227.26:4655 , piece_id=GQ3COM326RQ2EMLQMLP7FVVJORQ6PMARQ5Z5GMA4I3VLPKXF4BQQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.264826584 (92h20m50.092534978s) [noise=true , address=145.79.227.24:4581 , piece_id=5SSHQCN2EYGPLJVSKI44J4HZNQ3LQLQWBTQIRHNIG6NMHY3UIXPQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265392667 (92h20m50.093101058s) [noise=true , address=145.79.227.32:4836 , piece_id=EVN7KATZSO6NSNJTWVNBKX7UCH335NKHNATFMCGJ55SFJQGAQDAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265603132 (92h20m50.093311525s) [noise=true , address=145.79.227.31:4798 , piece_id=6QPX2IVROWBITNETPBV5AHEGSW45VMBLFCP67ASI5655CO446RUA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.265776710 (92h20m50.093485104s) [noise=true , address=145.79.227.14:4840 , piece_id=X3IK2RCFFER3TZZGA33GMELZNMW3KWGFRIFRM6ZFM5TISXRQOVDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266020403 (92h20m50.093728794s) [noise=true , address=145.79.227.41:4086 , piece_id=E32A6M6R35XNW37RTVNJWAQ7MPIXQ7SNW6N5S7IYWHZHUKBYZK3A, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.266141894 (92h20m50.093850288s) [noise=true , address=145.79.227.23:4545 , piece_id=FE2DPKR6JCDNWJYY22Q4T2P4GGH7U5TW6K44KEND5YA3EV6QFNAQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.273836124 (92h20m50.101544516s) [noise=true , address=145.79.227.16:4911 , piece_id=FXCLDL2E33RR4FS3FP5BCR6VEOQZ6CTUCKWPKVWEKDKIWQIBHBRA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:49.276400349 (92h20m50.10410874s) [noise=true , address=45.91.116.11:4119 , piece_id=5DCMCPJWHBKWQWHEG7MPIHLPM3H6DSRPY246USHNYP6DFDEBIMSA, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:48:51.800506085 (92h20m52.628214488s) [noise=true , address=145.79.227.44:4194 , piece_id=4CKSDREYCMWMTNNCOQEQWOCTHQPAXEVOATM5XQ46ZIQOOHVIQQ4Q, seg_pos={0 0}, upload=95998]: Done uploading piece. err: <nil> Dec 9 11:49:19.125708929 (92h21m19.953417331s) [noise=true , address=77.90.31.15:4698 , piece_id=FERGIZEYNZ3Q7ZZXFSZZ5M4PGKHQEAHUWLNYFTPBAYPGMFQDFXZQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125905800 (92h21m19.953614199s) [noise=true , address=77.90.31.16:4735 , piece_id=4JGZHZGLYDNJK7DHVSULD2I2HD5SIUFJLCRR36PEEEBSX2XSVMDQ, seg_pos={0 0}, upload=95998]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 11:49:19.125940842 (92h21m19.953649225s) [seg_pos={0 0}, upload=95998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 11:49:19.144876357 (92h21m19.97258475s) [upload=95998]: Done uploading segments... Dec 9 11:49:19.144916478 (92h21m19.972624869s) [upload=95998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.907054ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)