Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 470.646843ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:18:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.679589ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 497.172147ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.281856ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 497.291222ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:17:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.316624ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 442.276538ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.307642ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 536.383291ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:16:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.412516ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 469.66945ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:15:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.774204ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 521.372721ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:14:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.400093ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 454.500661ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:13:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.542172ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 505.150074ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:12:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.18113ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 482.61797ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:11:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.646603ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 534.444277ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:11:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 534.46587ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 544.497677ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:09:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.523606ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 529.321917ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:09:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.352056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 495.930809ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 495.970919ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 521.760806ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:08:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.794034ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 562.919234ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:07:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 562.94663ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 475.831782ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:06:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.862227ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 735.655487ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:06:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 735.67976ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 466.332608ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:05:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.449697ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 538.581527ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:04:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 538.608517ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 481.668704ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 481.700012ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 508.305771ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:02:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.322401ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 448.430632ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:01:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.456034ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 541.203134ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.229164ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 569.025111ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 569.040924ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 520.979945ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:58:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 521.069192ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 494.460959ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:57:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.489769ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 495.552096ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 495.586575ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 484.052214ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 484.082073ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 742.510868ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:55:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 742.54193ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 522.34843ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.379789ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 458.885655ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:54:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.915672ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 977.901067ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:53:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 977.964568ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.025124715s trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 21:53:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.025170844s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.048752427s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:54:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.048777321s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.054546449s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:53:44 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.054588692s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.667345744s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:53:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.667368877s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.140545407s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:52:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.140572815s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.483253939s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:52:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.483290208s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.088027647s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:52:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.088055721s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.565177003s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:51:43 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.565202445s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.054620408s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:51:22 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.05465247s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.581325967s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:50:45 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.581358058s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.492659311s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:50:35 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.492707697s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.578228349s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:50:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.578255604s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.465862182s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:49:26 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.465894073s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.613261625s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:49:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.613356036s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.564836181s trace: https://jaeger.dev.storj.io/trace/68bea6b4a9ab6dba FAIL (2025-12-22 16:48:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.564862826s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.582884358s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:48:27 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.582930881s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.530336251s trace: https://jaeger.dev.storj.io/trace/6a1ed17440f81050 FAIL (2025-12-22 16:48:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.530360165s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)