Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 22.714874ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Vary=Accept-Encoding Cf-Ray=9a92588deca98d3f-EWR Alt-Svc=h3=":443"; ma=86400 Set-Cookie=_cfuvid=Rx8foEFIwBP.w0p5KGHIwAM2YjswxznH_HLgTQ1vRHI-1764925150392-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Date=Fri, 05 Dec 2025 08:59:10 GMT Content-Type=text/html Content-Length=175 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6f3e362b12c6cefa FAIL (2025-12-05 08:59:10 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 22.734176ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 28.534723ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=FfAOwuJI1776eQNawMHt3v9dXMAzy2wzJ21wUMhZwW0-1764925127846-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a925800fa4d0ca6-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:58:47 GMT target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/56b7b5cad0a80c0f FAIL (2025-12-05 08:58:47 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.558183ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 26.042773ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:58:43 GMT Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=_vBGbqTXoJ0ddmwrKAqWbTTRxaMA_zlD_acK4q76nAo-1764925123650-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Content-Type=text/html Cf-Ray=9a9257e6b93e0f64-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7ee122f0e8906579 FAIL (2025-12-05 08:58:43 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 26.064903ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 21.171198ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Cf-Ray=9a9257100f9b9b29-EWR Date=Fri, 05 Dec 2025 08:58:09 GMT Set-Cookie=_cfuvid=We4AXgV7UzUbxjArRjPa4WzfzEncZEhTucS5wvmmjO8-1764925089293-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/184ffeffcc2dbf6f FAIL (2025-12-05 08:58:09 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 21.192659ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 34.493961ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:58:06 GMT Content-Length=175 Vary=Accept-Encoding Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Set-Cookie=_cfuvid=Zw5yjy3RKznoe_UbpQyKceuI1.me5x92cdgg6_3EFjE-1764925086533-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a9256fecbea0c86-EWR target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/64c01af50778af4 FAIL (2025-12-05 08:58:06 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 34.521569ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 14 out of 15 is failed (threshold=6) [.XXXXXXXXXXXXXX] duration: 24.372865ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Server=cloudflare Cf-Ray=9a92563e3a4ff3e6-EWR Date=Fri, 05 Dec 2025 08:57:35 GMT Vary=Accept-Encoding Set-Cookie=_cfuvid=M6BacFzQ7EZ03w_5spfoUF9VzU5uV2kCdPinrtJPBPk-1764925055720-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a348a242b46a1ab FAIL (2025-12-05 08:57:35 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.413126ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 21.153527ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Cf-Ray=9a925539cb5cc8c6-EWR Content-Type=text/html Set-Cookie=_cfuvid=aoPHnLcyZG8pwpEzvI_oSI4m9tuQwW1UxW4bTvdSKfg-1764925014054-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:56:54 GMT Content-Length=175 Vary=Accept-Encoding Server=cloudflare target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3e7726e717f4b5ce FAIL (2025-12-05 08:56:54 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 21.192752ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 15 out of 15 is failed (threshold=6) [XXXXXXXXXXXXXXX] duration: 32.487543ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:56:36 GMT Content-Length=175 Set-Cookie=_cfuvid=8ZLykFLZr4h__tUVuPjXjdBuBkwVKLu07uaMAjJ9VeI-1764924996197-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Content-Type=text/html Vary=Accept-Encoding Server=cloudflare Cf-Ray=9a9254ca1f22b1bc-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57e97a39d4fdcd77 FAIL (2025-12-05 08:56:36 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 32.503285ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 14 out of 15 is failed (threshold=6) [.XXXXXXXXXXXXXX] duration: 24.27932ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Length=175 Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:56:32 GMT Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=RUtY9KOqldrEsIhAuKzk2zr7hziEY8h1XLWrLX1Mwns-1764924992526-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a9254b33a8f7274-EWR target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/177f32b5d52b4e9b FAIL (2025-12-05 08:56:32 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.301293ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 13 out of 15 is failed (threshold=6) [..XXXXXXXXXXXXX] duration: 24.68246ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=01uVXVNPR9fCd7ofCZbSUPkpvoAvAHUtC1sENmwO1HE-1764924988322-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a925498f902c8c6-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:56:28 GMT Content-Type=text/html Vary=Accept-Encoding Server=cloudflare Content-Length=175 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/493260667a1e8bb7 FAIL (2025-12-05 08:56:28 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.693539ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 13 out of 15 is failed (threshold=6) [..XXXXXXXXXXXXX] duration: 33.320673ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Vary=Accept-Encoding Server=cloudflare Content-Length=175 Set-Cookie=_cfuvid=BeqsI2_6oPpHyq5M1PKDvcdjOl9_T35kL_RscJAphCY-1764924984715-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a9254825a9f55d7-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:56:24 GMT target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ccc4d352bf4db84 FAIL (2025-12-05 08:56:24 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 33.349747ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 12 out of 15 is failed (threshold=6) [...XXXXXXXXXXXX] duration: 22.79072ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Cf-Ray=9a92537d7d385a29-EWR Alt-Svc=h3=":443"; ma=86400 Content-Length=175 Vary=Accept-Encoding Server=cloudflare Date=Fri, 05 Dec 2025 08:55:42 GMT Content-Type=text/html Set-Cookie=_cfuvid=6l2T1sFOHYwv7n4iKehWm3RqPtxQ5D1wa104ziDaaO0-1764924942967-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/661ac66870cc4002 FAIL (2025-12-05 08:55:42 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 22.813502ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 11 out of 15 is failed (threshold=6) [....XXXXXXXXXXX] duration: 21.924ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=ueLXOqn81WC_2YnGjkNlWZr1UBFaVeboNPLAd7vbwKU-1764924914509-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a9252cbabfdd2b1-EWR Date=Fri, 05 Dec 2025 08:55:14 GMT Content-Length=175 Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/49de0bb6902caeb9 FAIL (2025-12-05 08:55:14 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 21.941211ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 12 out of 15 is failed (threshold=6) [...XXXXXXXXXXXX] duration: 27.781722ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=JlLXjCTtrXMx0_lBEHWskCHAtGH_jVFHGlHJ.YBVM8E-1764924911532-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:55:11 GMT Vary=Accept-Encoding Cf-Ray=9a9252b8ff61dafc-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7ae3f83249f75191 FAIL (2025-12-05 08:55:11 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.860109ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 10 out of 15 is failed (threshold=6) [.....XXXXXXXXXX] duration: 22.708707ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Date=Fri, 05 Dec 2025 08:54:57 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=65gTwi.xFgyBDIFvPr7UCAUUr9tQ.k14GXWu1da9fNM-1764924897905-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a925263dff7729c-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4c814affb6e22c7c FAIL (2025-12-05 08:54:57 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 22.718977ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 11 out of 15 is failed (threshold=6) [....XXXXXXXXXXX] duration: 27.536151ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=nlOeSDTar898Pls2I_Fjk7kYapYHHvOH.Ztj2uBHezs-1764924893907-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:54:53 GMT Content-Length=175 Server=cloudflare Cf-Ray=9a92524ada3942f2-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/b561b5d41a8e1a3 FAIL (2025-12-05 08:54:53 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.560444ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 9 out of 15 is failed (threshold=6) [......XXXXXXXXX] duration: 29.439414ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Cf-Ray=9a9251c64b9080dc-EWR Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=FbA13R_MsmhvH9HLElRn3JGu1366kDMx9CxNt2qiB30-1764924872699-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Date=Fri, 05 Dec 2025 08:54:32 GMT Content-Length=175 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4376699236109c9a FAIL (2025-12-05 08:54:32 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 29.46542ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 10 out of 15 is failed (threshold=6) [.....XXXXXXXXXX] duration: 47.110574ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=ILWZZV26WirsUqSF0L8hLETI0F03O3Ur8DtnKy0uU4M-1764924853110-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:54:13 GMT Vary=Accept-Encoding Cf-Ray=9a92514bda8175b1-EWR Content-Type=text/html Content-Length=175 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4a67c1d66f7b726b FAIL (2025-12-05 08:54:13 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 47.131953ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 8 out of 15 is failed (threshold=6) [.......XXXXXXXX] duration: 23.763371ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:53:34 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=XH6jsO6WPX8nM9.F_mWvhzmEzvjjRHs4yiPa9EERKbM-1764924814391-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a925059ed224232-EWR Alt-Svc=h3=":443"; ma=86400 Vary=Accept-Encoding target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57cf3042c71eef88 FAIL (2025-12-05 08:53:34 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 23.788194ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 9 out of 15 is failed (threshold=6) [......XXXXXXXXX] duration: 29.661979ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:53:23 GMT Content-Type=text/html Set-Cookie=_cfuvid=8ecuaxzPpsld5pAaDpRJ1ISkJF3Zrf98uqUHlGeoYG4-1764924803936-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Content-Length=175 Vary=Accept-Encoding Server=cloudflare Cf-Ray=9a9250188c9d0fa8-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/74187d996789cf8d FAIL (2025-12-05 08:53:23 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 29.686067ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 7 out of 15 is failed (threshold=6) [........XXXXXXX] duration: 26.882602ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:53:03 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Server=cloudflare Cf-Ray=9a924f99bbbfb35a-EWR Set-Cookie=_cfuvid=u9Pt1ziEdokDBn8nwPmGlCVtrXe9Pnz_F8JCxhHp3P0-1764924783647-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3b038c8b410efe00 FAIL (2025-12-05 08:53:03 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 26.919945ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 8 out of 15 is failed (threshold=6) [.......XXXXXXXX] duration: 24.75658ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:52:45 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=6A8IgG89o9ESMtayYAwah8JlNrVQjIiRbxR1pFJfjgE-1764924765627-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Cf-Ray=9a924f291afb58d7-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/466155f9d2e765e2 FAIL (2025-12-05 08:52:45 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.777501ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 6 out of 15 is failed (threshold=6) [.........XXXXXX] duration: 29.321529ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=EW2VDzn5qngFIga.l2E7KfItjzZ3cgokuRIN41y1fOg-1764924735022-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Server=cloudflare Cf-Ray=9a924e69da125e79-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:52:15 GMT target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/590b803a6f5351a2 FAIL (2025-12-05 08:52:15 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 29.346814ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 7 out of 15 is failed (threshold=6) [........XXXXXXX] duration: 27.667063ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:52:06 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Cf-Ray=9a924e34faed9608-EWR Alt-Svc=h3=":443"; ma=86400 Set-Cookie=_cfuvid=mfmctKwpg3N3DYeFpOpYNM8AmivvDOluQK7EqXt4ZDM-1764924726561-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/59154a2467230ae FAIL (2025-12-05 08:52:06 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.686227ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 6 out of 15 is failed (threshold=6) [.........XXXXXX] duration: 25.487475ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:51:27 GMT Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=QxQ4YR5SLmgpzh4LFa0ua6wBaQJtdc1MBkcH5VqQeNI-1764924687180-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Content-Type=text/html Server=cloudflare Cf-Ray=9a924d3ece4e0f7d-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/39a21b99b7e64e14 FAIL (2025-12-05 08:51:27 UTC) unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 25.513622ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 5 out of 15 is failed (threshold=6) [..........XXXXX] duration: 27.012064ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:51:14 GMT Content-Type=text/html Server=cloudflare Cf-Ray=9a924cef5911c094-EWR Alt-Svc=h3=":443"; ma=86400 Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=kb0s2JLRlx9ACcoP3gj3xzkF1Qd_QCCiax2yBVIViYs-1764924674464-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/422aa6a65312cc73 FAIL (2025-12-05 08:51:14 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.031424ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 4 out of 15 is failed (threshold=6) [...........XXXX] duration: 27.164817ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:50:55 GMT Content-Type=text/html Content-Length=175 Server=cloudflare Cf-Ray=9a924c7aaee842c4-EWR Alt-Svc=h3=":443"; ma=86400 Vary=Accept-Encoding Set-Cookie=_cfuvid=Ag4xuRAQHLpeQudrZ3DYfTwh3ae5IbfmQt392GU3gjk-1764924655800-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/23f6323e4a3078d0 FAIL (2025-12-05 08:50:55 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.186297ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 5 out of 15 is failed (threshold=6) [..........XXXXX] duration: 28.705256ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=brm3ivDgnSlBen5MkA9B9Ns7dXvYn_QPaXhnEJBPziw-1764924654999-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924c75ab5a41d9-EWR Vary=Accept-Encoding Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:50:54 GMT target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/17a91e996401fe7d FAIL (2025-12-05 08:50:54 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.732114ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 3 out of 15 is failed (threshold=6) [............XXX] duration: 27.579589ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/html Vary=Accept-Encoding Set-Cookie=_cfuvid=IMsL7uZyYyC65VJGCKOq_KXWrLIYgLDS3MwLcA4u4bc-1764924624997-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924bba2bae985c-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:50:24 GMT Content-Length=175 Server=cloudflare target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/53889cc3c5aeb143 FAIL (2025-12-05 08:50:24 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.601632ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 4 out of 15 is failed (threshold=6) [...........XXXX] duration: 27.224651ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:50:03 GMT Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=dKHsIDeqVKGXiVbPs0rLmzxLTA7uD1r4gv054q6xquU-1764924603784-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 Content-Type=text/html Server=cloudflare Cf-Ray=9a924b35999c0f9b-EWR target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/55659b4b99405ee4 FAIL (2025-12-05 08:50:03 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 27.260723ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 3 out of 15 is failed (threshold=6) [............XXX] duration: 28.411584ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Cf-Ray=9a924ac4390419c7-EWR Date=Fri, 05 Dec 2025 08:49:45 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=51AmdQjM6l20eyhm80QaTpHK5JMB9mwcdWtx7WLIK7w-1764924585643-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2f074165b761603a FAIL (2025-12-05 08:49:45 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.435463ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [.............XX] duration: 24.031597ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 05 Dec 2025 08:49:43 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=sOyyajCy3TmxFUOMwHdykkIs7tG.6ConBgSiIMU2R38-1764924583207-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a924ab4f9a7610c-EWR Server=cloudflare Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/45ad4bfa4c82b7d1 FAIL (2025-12-05 08:49:43 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 24.057741ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.............XX] duration: 28.39113ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=r9Bo.M9DdlqZbfUYhjpiNbHlZMQhTbVfRkQJ4IOREBU-1764924526044-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a92494fbd390f8c-EWR Alt-Svc=h3=":443"; ma=86400 Date=Fri, 05 Dec 2025 08:48:46 GMT Content-Length=175 Server=cloudflare Content-Type=text/html Vary=Accept-Encoding target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/ea5ff2961c4037 FAIL (2025-12-05 08:48:46 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 28.411825ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 33.163658ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Set-Cookie=_cfuvid=MNDMKtEGkEHqx5teuOTcK73tKV4J.jPayzp_0ot0O.k-1764924509164-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Alt-Svc=h3=":443"; ma=86400 Vary=Accept-Encoding Cf-Ray=9a9248e62ce24380-EWR Date=Fri, 05 Dec 2025 08:48:29 GMT Content-Type=text/html Content-Length=175 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/397cfa9fd6f20042 FAIL (2025-12-05 08:48:29 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 33.183162ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 25.756727ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Server=cloudflare Date=Fri, 05 Dec 2025 08:48:19 GMT Content-Type=text/html Content-Length=175 Vary=Accept-Encoding Set-Cookie=_cfuvid=oqlvgD3ZKx.ZDU7PRv3tOiY63STnoWV97xHnmRWoU30-1764924499637-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Cf-Ray=9a9248aaacf0423d-EWR Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1fe6f67af673aa6f FAIL (2025-12-05 08:48:19 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 25.779304ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 595.987548ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 596.021225ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 575.349403ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:56:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 575.379708ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 571.31928ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:55:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.360474ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 493.494686ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:54:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.522647ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 545.542131ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 545.570383ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 505.061918ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:52:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.088154ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 585.710652ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:51:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 585.750181ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.1916498s trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:51:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.191678854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 573.977624ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 574.024232ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 463.362189ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:49:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.457971ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 472.960408ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 472.98392ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 551.070147ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 551.099796ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 555.655035ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 555.701745ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 459.178458ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:46:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.205093ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 447.611525ms trace: https://jaeger.dev.storj.io/trace/187b00ea59b65c93 FAIL (2025-12-04 18:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 4 18:08:44.094209764 (560h22m8.861893189s) [upload=583262]: Starting upload Dec 4 18:08:44.094287136 (560h22m8.861970551s) [upload=583262]: Done starting upload Dec 4 18:08:44.094461765 (560h22m8.862145191s) [upload=583262]: Uploading segments... Dec 4 18:08:44.094920739 (560h22m8.862604165s) [upload=583262]: Got next segment. Inline: false Dec 4 18:08:44.094974933 (560h22m8.862658358s) [upload=583262]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 18:08:44.094989931 (560h22m8.862673356s) [upload=583262]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 18:08:45.100410605 (560h22m9.868094042s) [seg_pos={0 0}, upload=583262]: Begin upload segment... Dec 4 18:08:45.100476548 (560h22m9.868159975s) [seg_pos={0 0}, upload=583262]: Done begin upload segment. Dec 4 18:08:45.100529778 (560h22m9.868213207s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.100539715 (560h22m9.868223151s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101188254 (560h22m9.868871689s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101422177 (560h22m9.869105613s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.101897607 (560h22m9.86958104s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102329282 (560h22m9.870012716s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.102752764 (560h22m9.870436199s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.103281664 (560h22m9.870965099s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.104488607 (560h22m9.872172042s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.106036461 (560h22m9.873719895s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.107284781 (560h22m9.874968212s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.109667473 (560h22m9.877350904s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.112625924 (560h22m9.880309358s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.114900649 (560h22m9.882584083s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.121399785 (560h22m9.889083219s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.123860346 (560h22m9.891543764s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Uploading piece... Dec 4 18:08:45.147197792 (560h22m9.914881223s) [noise=true , address=77.90.31.19:4841 , piece_id=KVU4B764TEEEENGTM26HMWVOPKGWZHVFI2HRD3Q33UT7GMVDVTBA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.185546035 (560h22m9.95322946s) [upload=583262]: Next returned nil segment Dec 4 18:08:45.185558346 (560h22m9.953241778s) [upload=583262]: Waiting for error group managing segments... Dec 4 18:08:45.194314196 (560h22m9.961997621s) [noise=true , address=77.90.31.13:4627 , piece_id=BHLPRTMYCR4U6YJJIJESGSLLIBWTMJXIGIPWIXRLGLCXHSMZ7PKQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194723371 (560h22m9.962406802s) [noise=true , address=145.79.227.26:4653 , piece_id=BGDICVNG4N3HAZNUBTBMKR7O5SLFJJTIHDYXD5DPJWRS4DJP2J4Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.194874971 (560h22m9.962558397s) [noise=true , address=145.79.227.44:4195 , piece_id=QOAYDYAYUJ72HQZW5JJWERVYRRLOTUTEAYH7MHXVFOFDZ4IRJVDA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195694670 (560h22m9.963378101s) [noise=true , address=145.79.227.14:4839 , piece_id=DAPOABOT4FVPEJWFVWISFB2L3U7IOS52YI73KRXL72SPFI5E6F3Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.195868147 (560h22m9.963551623s) [noise=true , address=145.79.227.27:4690 , piece_id=Q3GJITNZEWIT5UUCXAQTQ3PYJNEE62L4QEPCPCZXR4XBNEQXXVVQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.196138207 (560h22m9.963821633s) [noise=true , address=145.79.227.41:4085 , piece_id=ZGUQIITEYDQEJV5PR4PFQT3JIG2J4IXSBVRSFLGBDEATZ7DZ2EHQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.197113442 (560h22m9.964796872s) [noise=true , address=145.79.227.43:4157 , piece_id=725SSNYPLCIB6ASUI4LOYGDITUGALXERNNFAVV4SU7ZE25SB22AQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.198903071 (560h22m9.9665865s) [noise=true , address=145.79.227.32:4836 , piece_id=NJWVO4WDDO3LRRJXZL42ZYPMMLOSPTB3YXAJSQ4LF2W6JRH7XEZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.199173870 (560h22m9.966857296s) [noise=true , address=145.79.227.38:4977 , piece_id=6SZ4G3MC4Q6KJVK572UWSEU23AZDFTAH2XMITZ52FPW7ZZF5ALZQ, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200091557 (560h22m9.96777499s) [noise=true , address=145.79.227.40:4049 , piece_id=RAT5ADZIR23NDUKNME3L62OHDB5BGTRGNZJUC46NTTHFV2TBVS2A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.200834039 (560h22m9.968517472s) [noise=true , address=77.90.31.17:4771 , piece_id=MWGGSJ43UJA2TP5RFZ2R3HEFRSCEVEJBKJFN6VYUOLFV37PXTKYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:08:45.202648776 (560h22m9.970332204s) [noise=true , address=145.79.227.30:4763 , piece_id=FQLQSDHH5DU6ZO2WQBVTUQJFWJDHBHXWEOAOPD4YSUJCMDM3FI6A, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 4 18:08:45.315871915 (560h22m10.083555346s) [noise=true , address=77.90.31.18:4805 , piece_id=CMVEH2HK3MTPMKJHVHOXTYGNYHSZ4I5JQUFFR5QP2WSNADYRBUYA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: <nil> Dec 4 18:09:14.061860489 (560h22m38.829543924s) [noise=true , address=77.90.31.16:4734 , piece_id=AZKSICIL5DQT4NZ5XT2J4ROSXAPMAL5LVOSWD7265DDAFRRVMCHA, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061930233 (560h22m38.829613665s) [noise=true , address=77.90.31.12:4590 , piece_id=J3FN3B5SJZXPFL7MLCP4JUZH5PIXC4SFJLPSXMCP5MA2B7F3BD2Q, seg_pos={0 0}, upload=583262]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 4 18:09:14.061982745 (560h22m38.829666165s) [seg_pos={0 0}, upload=583262]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 4 18:09:14.094344994 (560h22m38.86202842s) [upload=583262]: Done uploading segments... Dec 4 18:09:14.094378677 (560h22m38.862062102s) [upload=583262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.637868ms core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)