Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 8 out of 15 is failed (threshold=6) [.......XXXXXXXX] duration: 24.75658ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:52:45 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=6A8IgG89o9ESMtayYAwah8JlNrVQjIiRbxR1pFJfjgE-1764924765627-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Cf-Ray=9a924f291afb58d7-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/466155f9d2e765e2 FAIL (2025-12-05 08:52:45 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.777501ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 6 out of 15 is failed (threshold=6) [.........XXXXXX] duration: 29.321529ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=EW2VDzn5qngFIga.l2E7KfItjzZ3cgokuRIN41y1fOg-1764924735022-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Server=cloudflare Cf-Ray=9a924e69da125e79-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:52:15 GMT target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/590b803a6f5351a2 FAIL (2025-12-05 08:52:15 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 29.346814ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 7 out of 15 is failed (threshold=6) [........XXXXXXX] duration: 27.667063ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:52:06 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Cf-Ray=9a924e34faed9608-EWR Alt-Svc=h3=":443"; ma=86400 Set-Cookie=_cfuvid=mfmctKwpg3N3DYeFpOpYNM8AmivvDOluQK7EqXt4ZDM-1764924726561-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/59154a2467230ae FAIL (2025-12-05 08:52:06 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.686227ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 6 out of 15 is failed (threshold=6) [.........XXXXXX] duration: 25.487475ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:51:27 GMT Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=QxQ4YR5SLmgpzh4LFa0ua6wBaQJtdc1MBkcH5VqQeNI-1764924687180-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Content-Type=text/html Server=cloudflare Cf-Ray=9a924d3ece4e0f7d-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/39a21b99b7e64e14 FAIL (2025-12-05 08:51:27 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 25.513622ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 5 out of 15 is failed (threshold=6) [..........XXXXX] duration: 27.012064ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:51:14 GMT Content-Type=text/html Server=cloudflare Cf-Ray=9a924cef5911c094-EWR Alt-Svc=h3=":443"; ma=86400 Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=kb0s2JLRlx9ACcoP3gj3xzkF1Qd_QCCiax2yBVIViYs-1764924674464-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/422aa6a65312cc73 FAIL (2025-12-05 08:51:14 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.031424ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 4 out of 15 is failed (threshold=6) [...........XXXX] duration: 27.164817ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:50:55 GMT Content-Type=text/html Content-Length=175 Server=cloudflare Cf-Ray=9a924c7aaee842c4-EWR Alt-Svc=h3=":443"; ma=86400 Vary=Accept-Encoding Set-Cookie=_cfuvid=Ag4xuRAQHLpeQudrZ3DYfTwh3ae5IbfmQt392GU3gjk-1764924655800-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/23f6323e4a3078d0 FAIL (2025-12-05 08:50:55 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.186297ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 5 out of 15 is failed (threshold=6) [..........XXXXX] duration: 28.705256ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=brm3ivDgnSlBen5MkA9B9Ns7dXvYn_QPaXhnEJBPziw-1764924654999-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924c75ab5a41d9-EWR Vary=Accept-Encoding Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:50:54 GMT target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/17a91e996401fe7d FAIL (2025-12-05 08:50:54 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.732114ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 3 out of 15 is failed (threshold=6) [............XXX] duration: 27.579589ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=IMsL7uZyYyC65VJGCKOq_KXWrLIYgLDS3MwLcA4u4bc-1764924624997-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924bba2bae985c-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:50:24 GMT Content-Length=175 Server=cloudflare target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/53889cc3c5aeb143 FAIL (2025-12-05 08:50:24 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.601632ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 4 out of 15 is failed (threshold=6) [...........XXXX] duration: 27.224651ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:50:03 GMT Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=dKHsIDeqVKGXiVbPs0rLmzxLTA7uD1r4gv054q6xquU-1764924603784-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Server=cloudflare Cf-Ray=9a924b35999c0f9b-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/55659b4b99405ee4 FAIL (2025-12-05 08:50:03 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.260723ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 3 out of 15 is failed (threshold=6) [............XXX] duration: 28.411584ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Cf-Ray=9a924ac4390419c7-EWR Date=Fri, 05 Dec 2025 08:49:45 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=51AmdQjM6l20eyhm80QaTpHK5JMB9mwcdWtx7WLIK7w-1764924585643-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2f074165b761603a FAIL (2025-12-05 08:49:45 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.435463ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [.............XX] duration: 24.031597ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:49:43 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=sOyyajCy3TmxFUOMwHdykkIs7tG.6ConBgSiIMU2R38-1764924583207-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924ab4f9a7610c-EWR Server=cloudflare Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/45ad4bfa4c82b7d1 FAIL (2025-12-05 08:49:43 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.057741ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.............XX] duration: 28.39113ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=r9Bo.M9DdlqZbfUYhjpiNbHlZMQhTbVfRkQJ4IOREBU-1764924526044-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a92494fbd390f8c-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:48:46 GMT Content-Length=175 Server=cloudflare Content-Type=text/html Vary=Accept-Encoding target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ea5ff2961c4037 FAIL (2025-12-05 08:48:46 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.411825ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 33.163658ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=MNDMKtEGkEHqx5teuOTcK73tKV4J.jPayzp_0ot0O.k-1764924509164-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Vary=Accept-Encoding Cf-Ray=9a9248e62ce24380-EWR Date=Fri, 05 Dec 2025 08:48:29 GMT Content-Type=text/html Content-Length=175 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/397cfa9fd6f20042 FAIL (2025-12-05 08:48:29 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 33.183162ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 25.756727ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Date=Fri, 05 Dec 2025 08:48:19 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=oqlvgD3ZKx.ZDU7PRv3tOiY63STnoWV97xHnmRWoU30-1764924499637-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a9248aaacf0423d-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1fe6f67af673aa6f FAIL (2025-12-05 08:48:19 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 25.779304ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 595.987548ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 596.021225ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 575.349403ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:56:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 575.379708ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 571.31928ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:55:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.360474ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 493.494686ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:54:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.522647ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 545.542131ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 545.570383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 505.061918ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:52:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.088154ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 585.710652ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:51:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 585.750181ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.1916498s trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:51:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.191678854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 573.977624ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 574.024232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 463.362189ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:49:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.457971ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 472.960408ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.98392ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 551.070147ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 551.099796ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 555.655035ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 555.701745ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 459.178458ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.205093ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 447.611525ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.637868ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 500.647726ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:45:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 500.846303ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 571.27008ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:44:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.288408ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 453.190492ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.225588ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 553.198103ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.224736ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 548.744921ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:41:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 548.771778ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 501.594425ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:40:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.640201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 485.116303ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.232959ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 433.942539ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:39:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.983327ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 452.780038ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:38:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 452.903645ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 471.444018ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.472575ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 494.58813ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:36:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.617298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 539.487165ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:35:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 539.513407ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 550.68408ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 550.769914ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 466.395707ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:34:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.48439ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 606.024316ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:33:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 606.069459ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 652.607184ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:33:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 652.635852ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 477.407392ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:32:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.468675ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 482.1465ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 482.175305ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 541.070879ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:30:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.1011ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 472.45745ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.483434ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 512.111495ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:29:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.137722ms core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)