Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 2.783115309s trace: https://jaeger.dev.storj.io/trace/48a956946c09c203 FAIL (2025-12-15 11:50:31 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881613A457239B2, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 2.783206835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................................X.XX] duration: 4.560589368s trace: https://jaeger.dev.storj.io/trace/4bda253a981ceed1 FAIL (2025-12-15 11:50:09 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881613511C801C8, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 4.560695294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 2.11772997s trace: https://jaeger.dev.storj.io/trace/73a41d0b67115b79 FAIL (2025-12-15 11:49:56 UTC) operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881613200B5B080, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 2.11783788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 3.739468914s trace: https://jaeger.dev.storj.io/trace/167144d49fc9e629 FAIL (2025-12-15 11:49:28 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881612B9232CA7D, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 3.739569452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................X.X] duration: 3.019715054s trace: https://jaeger.dev.storj.io/trace/34b09c35c976f88a FAIL (2025-12-15 11:49:05 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 188161262B735BEC, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 3.019741617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 418.75227ms trace: https://jaeger.dev.storj.io/trace/f6e008cef3a1e43 FAIL (2025-12-15 11:48:43 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816116C51ABFDE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 419.155886ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5.823272228s trace: https://jaeger.dev.storj.io/trace/21d6dbdf15da3c73 FAIL (2025-12-15 11:48:23 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1881611C6A68916A, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 11)) Duration: 5.823298431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 1.920081818s trace: https://jaeger.dev.storj.io/trace/f6e008cef3a1e43 FAIL (2025-12-15 11:47:59 UTC) orbiter warning: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 18816116C51ABFDE, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: failed to upload enough pieces (needed at least 14 but got 10)) Duration: 1.920109984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 609.833543ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:20:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 609.942449ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 606.519778ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 606.547348ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 373.318739ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 373.557705ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 481.602997ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:18:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 481.630436ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 454.466738ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.591324ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 885.784639ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:17:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 885.815746ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 510.239579ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:17:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.267532ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 581.624747ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 581.757353ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 713.527333ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:15:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 713.551424ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 455.234908ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:14:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.267892ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 562.540565ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 562.58804ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 393.403386ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:12:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 393.427393ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 488.361113ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 488.389911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 455.84881ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:10:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.886151ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 716.901587ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:10:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 716.927114ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 454.696274ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:09:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.725468ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 399.02164ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:08:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.046259ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 476.062133ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:07:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.164102ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 406.90913ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 406.954185ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 547.297733ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:05:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.344523ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 432.859515ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:05:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.887707ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 476.250558ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.277692ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 488.344201ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:03:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 488.3729ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 518.320245ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.377152ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 435.940911ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:01:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 435.969353ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 431.27226ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:01:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.3037ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 404.158209ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.183524ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 471.644716ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 23:00:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.671489ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.285165806s trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:59:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.285190882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 450.280694ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:58:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.376395ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 420.926275ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:57:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 421.227144ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 510.32421ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:56:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.355997ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 388.059393ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.085753ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 407.414614ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.446203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 450.400855ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:53:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.534371ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 399.66691ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:52:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.694108ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 480.018859ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:52:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.059046ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 420.093403ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:51:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.131259ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 455.754402ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:50:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.796011ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 490.384622ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:49:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 490.405699ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 639.171218ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:48:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 639.235007ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 487.383607ms trace: https://jaeger.dev.storj.io/trace/233043e29fa8c343 FAIL (2025-12-14 22:47:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 22:33:46.857315921 (223h5m47.685024323s) [upload=231966]: Starting upload Dec 14 22:33:46.857406697 (223h5m47.685115137s) [upload=231966]: Done starting upload Dec 14 22:33:46.857554892 (223h5m47.685263291s) [upload=231966]: Uploading segments... Dec 14 22:33:46.857789809 (223h5m47.685498211s) [upload=231966]: Got next segment. Inline: false Dec 14 22:33:46.857854892 (223h5m47.685563283s) [upload=231966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 22:33:46.857861581 (223h5m47.685569972s) [upload=231966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 22:33:46.883181005 (223h5m47.710889383s) [seg_pos={0 0}, upload=231966]: Begin upload segment... Dec 14 22:33:46.883263487 (223h5m47.710971938s) [seg_pos={0 0}, upload=231966]: Done begin upload segment. Dec 14 22:33:46.883341357 (223h5m47.71104976s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884115747 (223h5m47.711824152s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884624624 (223h5m47.712333022s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.884680897 (223h5m47.712389298s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885186841 (223h5m47.712895238s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885275901 (223h5m47.712984302s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885667114 (223h5m47.713375512s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.885870007 (223h5m47.713578389s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886162813 (223h5m47.713871216s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886397673 (223h5m47.714106075s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886906783 (223h5m47.714615185s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.886973316 (223h5m47.714681716s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887462068 (223h5m47.715170469s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887572123 (223h5m47.715280517s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.887988097 (223h5m47.71569651s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.888041031 (223h5m47.71574943s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Uploading piece... Dec 14 22:33:46.925527510 (223h5m47.753235912s) [noise=true , address=77.90.31.13:4626 , piece_id=PRCUKRX7ENP6YKDEOIDSV4GMH5Q5K7HVC4ZNZERBDHCER2QUNV4A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.943379059 (223h5m47.771087459s) [noise=true , address=77.90.31.16:4734 , piece_id=6TB524DIOSDWGFQMVAYE3AR65CVZ5CNMGOU2LWOFNNM6KNTNMMRA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.944339095 (223h5m47.772047493s) [noise=true , address=77.90.31.12:4589 , piece_id=33WO7DM4OWFEMY7PMTTHVGHPPJKIVLOWFWDJC4ENOPNG4RGJVVPA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 22:33:46.975134078 (223h5m47.802842461s) [upload=231966]: Next returned nil segment Dec 14 22:33:46.975152667 (223h5m47.802861058s) [upload=231966]: Waiting for error group managing segments... Dec 14 22:33:46.984199460 (223h5m47.811907862s) [noise=true , address=145.79.227.50:4373 , piece_id=A7AGWKNSGUQW4QL7FZW7VA6DRPP6BSFPWE7R4M7QBBQ2XUQAPAJQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.984521383 (223h5m47.812229782s) [noise=true , address=145.79.227.42:4123 , piece_id=WUZT2YMWJ57BTLL4PRAMWM5RMX4VQQPI3XRZ5XRFLAN2YN5IFCHQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.985430877 (223h5m47.813139272s) [noise=true , address=145.79.227.39:4013 , piece_id=F55VINDA232CQRSZPY5A46SKJDZFII67SXG7PUUL57JISXEZWQWQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.986245610 (223h5m47.813954001s) [noise=true , address=145.79.227.52:4445 , piece_id=TOBWVNBINBYYC2IUVBMZLEFVCC6PQ3IXCOBBT7SJZCGYERQOK2NQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.987118111 (223h5m47.814826507s) [noise=true , address=145.79.227.7:4293 , piece_id=7H2C64V2NK6B44LOZAJLO3SJK3FLSVDG3OLTQ3PLNVX2FR22ATTA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.990531453 (223h5m47.81823985s) [noise=true , address=145.79.227.28:4725 , piece_id=IUQKFFCWXWA3LT27L3TF3MALL3QQEZUEHGDJJRD7NUDU2AAJDVFQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.993914365 (223h5m47.821622758s) [noise=true , address=145.79.227.51:4410 , piece_id=I7MK4JNESUDE5KHSUDSLWWNX2TKFSARSPRRYFXJCJYPKCXLYT47A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.996262194 (223h5m47.823970588s) [noise=true , address=145.79.227.35:4481 , piece_id=6IODHBBUUA6EKJIT5XKESLSDD7J5BH7KZAHMCDI265Q6YTX22ZXQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998486391 (223h5m47.826194772s) [noise=true , address=145.79.227.40:4049 , piece_id=BJE7BEPTS54SK4TCW4O7LJM7473JX4354TAWCXI6XBSEDON3M7HQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.998859134 (223h5m47.826567535s) [noise=true , address=145.79.227.51:4409 , piece_id=V66MIUF6YEEWJ5I3A6SEDGWKXMGCLGQG5ORI7RJYKTHK44PVDARQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:46.999565357 (223h5m47.827273758s) [noise=true , address=145.79.227.29:4941 , piece_id=TARVAJFNJU4ESRKSHD2QJXK6RSAPAVIML4GKPSFSOELBOO6FYQZA, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:33:49.527532944 (223h5m50.355241325s) [noise=true , address=145.79.227.44:4194 , piece_id=PHUUOFF3MTLVU5NFNFQWBS4VH2KRIZTETS3OONR3VTGD242J4UTQ, seg_pos={0 0}, upload=231966]: Done uploading piece. err: <nil> Dec 14 22:34:16.824339067 (223h6m17.652047462s) [noise=true , address=77.90.31.19:4841 , piece_id=WGYTMINEAQY3PKMYDGTUQF7YE2TIIPLUBX5JZUN4REAWK5FSWB7A, seg_pos={0 0}, upload=231966]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 159.223.143.39:57630->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 22:34:16.824378696 (223h6m17.652087081s) [seg_pos={0 0}, upload=231966]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 22:34:16.843102183 (223h6m17.670810563s) [upload=231966]: Done uploading segments... Dec 14 22:34:16.843158344 (223h6m17.670866733s) [upload=231966]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.415974ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)