Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 453.463535ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.581506ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 470.893629ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.930145ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 430.796055ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:07:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.847044ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 464.801191ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:06:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.815893ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 482.526438ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.103484ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 475.465858ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:05:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 475.492203ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 497.619581ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:05:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.883971ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 562.794808ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:04:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.822626ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 623.342178ms trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:03:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 623.361468ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.00131386s trace: https://jaeger.dev.storj.io/trace/4718e63560644445 FAIL (2025-12-09 16:02:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001336113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 398.93788ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:26:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 398.966166ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 521.808484ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:25:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.83819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 454.699087ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:23:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.734983ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 505.888926ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:23:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.905413ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 560.794869ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:22:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 560.864005ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 489.445058ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:21:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 489.471976ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 494.195921ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:20:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.218798ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 471.08053ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.109362ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 574.857799ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 574.881042ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 478.296898ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:18:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.355625ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 457.503057ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:17:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.614781ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 457.842025ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:16:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.864909ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 561.929844ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 561.966432ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 481.796475ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 481.827811ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 470.883922ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.232181ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 404.393913ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:13:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.489695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 425.718724ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 425.786518ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 479.024238ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:11:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.065045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 455.562839ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.590406ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 455.260245ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.286881ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 568.365386ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.450763ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 499.870873ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:08:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.905781ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 455.848074ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:07:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.979949ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 475.075661ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:06:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.150882ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 509.280459ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 509.310247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 460.261272ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.290873ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 521.019513ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.051231ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 514.240455ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:03:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.284017ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 503.66699ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:01:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.698383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 449.141063ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.172974ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 427.509728ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:00:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.538816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 501.873127ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 13:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.929193ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 527.076382ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 527.106144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 465.776124ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:59:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.818614ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 400.848758ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:58:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 400.875085ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 455.940537ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:57:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.966924ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 566.386383ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:57:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 566.433168ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 535.180569ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 535.263046ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 552.292185ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 552.315906ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 417.075593ms trace: https://jaeger.dev.storj.io/trace/479b7c5ab37e1bfd FAIL (2025-12-09 12:54:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:39:24.868511635 (93h11m25.696220027s) [upload=96847]: Starting upload Dec 9 12:39:24.868604380 (93h11m25.696312763s) [upload=96847]: Done starting upload Dec 9 12:39:24.868784357 (93h11m25.69649275s) [upload=96847]: Uploading segments... Dec 9 12:39:24.869346996 (93h11m25.697055389s) [upload=96847]: Got next segment. Inline: false Dec 9 12:39:24.869395763 (93h11m25.697104154s) [upload=96847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:39:24.869400712 (93h11m25.697109106s) [upload=96847]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:39:24.914701948 (93h11m25.742410339s) [seg_pos={0 0}, upload=96847]: Begin upload segment... Dec 9 12:39:24.914755821 (93h11m25.742464213s) [seg_pos={0 0}, upload=96847]: Done begin upload segment. Dec 9 12:39:24.914795631 (93h11m25.742504027s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915166219 (93h11m25.742874615s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.915825055 (93h11m25.743533446s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916146395 (93h11m25.743854786s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916395121 (93h11m25.744103515s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916495745 (93h11m25.744204136s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916859392 (93h11m25.744567786s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.916941278 (93h11m25.744649678s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917188841 (93h11m25.744897232s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.917544274 (93h11m25.745252675s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918077147 (93h11m25.745785544s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.918561527 (93h11m25.746269919s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.919118467 (93h11m25.746826849s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.920557795 (93h11m25.748266189s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.922335682 (93h11m25.750044082s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.932985614 (93h11m25.760693999s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Uploading piece... Dec 9 12:39:24.955771386 (93h11m25.783479786s) [noise=true , address=77.90.31.15:4698 , piece_id=EJNQ3K5UYKYXCJCXE4SHLTARRVNJY553VT6QZISZW3HQYXZTKC7A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:24.956129193 (93h11m25.783837591s) [noise=true , address=77.90.31.12:4590 , piece_id=EOKJEZFAGQBRSVNB2VZJZ7PXMBULUEBHJMYSX5JDB5WPGANMTT5A, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 9 12:39:25.001945241 (93h11m25.829653633s) [upload=96847]: Next returned nil segment Dec 9 12:39:25.001957916 (93h11m25.829666307s) [upload=96847]: Waiting for error group managing segments... Dec 9 12:39:25.010917113 (93h11m25.838625512s) [noise=true , address=77.90.31.18:4807 , piece_id=YS5KKPCTSH52DS2JMZ3SBWFRCPJWYAA5DWKCRX2JXHRO4YLO2GNQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.011902376 (93h11m25.83961077s) [noise=true , address=145.79.227.9:4367 , piece_id=BWYDT62CD6XA6PXOCQXJDDC3JTWGURAJ5NLS3TR7JSF5T3MSLKIA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012450318 (93h11m25.840158709s) [noise=true , address=145.79.227.29:4942 , piece_id=BVQOF4GBOIFFZKG4RU5RWYOGZVDDPTSAFUU7RTUVDX7J2NZL6SCQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012513216 (93h11m25.840221608s) [noise=true , address=145.79.227.41:4086 , piece_id=VK5GUDMQTIH3D7PLPZSPOW6ZPXLEPQIEXH7QDM5SGAKQX7IG4ZHQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.012870568 (93h11m25.840579015s) [noise=true , address=145.79.227.42:4122 , piece_id=DOC73XIUJSM3XA4KT6N6AGYWPQ6DSYOHGA3OYR5IQTKMLXPLDNTA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013047973 (93h11m25.840756367s) [noise=true , address=145.79.227.24:4582 , piece_id=MLFJ2Q2TE7EG4GPSAEGA3SJ7YC4RUCC3E3YQ4ERGGGMKKBHANJEA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013201626 (93h11m25.840910018s) [noise=true , address=145.79.227.43:4157 , piece_id=JRKWYGWY6YHNIOBUPO2HKO277VHOZWIMJBLFDXSJFB4OZCMKFUQQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.013778594 (93h11m25.841486986s) [noise=true , address=145.79.227.32:4834 , piece_id=BPLTYRF6UFUBRRN6ITPAH75CRBG77POKA7AQ6SQKRTYUCXRRP6IA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.015742190 (93h11m25.843450581s) [noise=true , address=145.79.227.31:4797 , piece_id=NPALWD52IQH4OHXHECLLPSV277BTEYEX6OIMAXCIJHDWGYBSDLPQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.024438402 (93h11m25.852146794s) [noise=true , address=145.79.227.21:4473 , piece_id=W5S4S5PM3AVYM7Q76IL7HKWCZHIXS3DAUDS6TJCBSYFSSO62VLFA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.050939116 (93h11m25.87864751s) [noise=true , address=77.90.31.11:4553 , piece_id=GATJFIYFQAPBWHQQGKOESTKM7TT26IGACCSTAA3KXHXRF7UP5BZA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:25.274390056 (93h11m26.102098452s) [noise=true , address=77.90.31.21:4878 , piece_id=JOKSQMVG6A3QILGZQ3NC7S3ZCUP2KMCIDPXEU7WWE54ZQKLWMLIQ, seg_pos={0 0}, upload=96847]: Done uploading piece. err: <nil> Dec 9 12:39:54.838865093 (93h11m55.666573493s) [noise=true , address=77.90.31.19:4842 , piece_id=MU72KDT55HJ5NZIJO44ES7TS4KHMDXBTCTYSAHQEF5E34K3VD36Q, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838908446 (93h11m55.666616845s) [noise=true , address=77.90.31.16:4734 , piece_id=U7VPKLMAWHH3AGINEC2O5UTTOD3XOBCP7SIODOTDVEUW6NAZX5LA, seg_pos={0 0}, upload=96847]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:39:54.838986978 (93h11m55.666695372s) [seg_pos={0 0}, upload=96847]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:39:54.862701987 (93h11m55.690410381s) [upload=96847]: Done uploading segments... Dec 9 12:39:54.862766061 (93h11m55.690474453s) [upload=96847]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.106063ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)