Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 83.843535ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:07:52 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 83.922301ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXXXXXXX] duration: 5.001344996s trace: https://jaeger.dev.storj.io/trace/292ba761124170c3 FAIL (2026-03-18 21:07:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001371398s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 93.528981ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:07:07 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 93.580052ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXXXXXX] duration: 5.001608042s trace: https://jaeger.dev.storj.io/trace/21616c5b7dc078f5 FAIL (2026-03-18 21:06:58 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001625713s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 85.958949ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:06:49 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 86.093461ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXXXXX] duration: 5.001061362s trace: https://jaeger.dev.storj.io/trace/15992d06d493ba25 FAIL (2026-03-18 21:06:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001109031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXXXX] duration: 5.001137242s trace: https://jaeger.dev.storj.io/trace/44e46a2add3faab0 FAIL (2026-03-18 21:06:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001150593s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 96.632469ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:05:58 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 96.666037ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXXX] duration: 5.001969312s trace: https://jaeger.dev.storj.io/trace/f25fd3b41dc54f9 FAIL (2026-03-18 21:05:41 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002000158s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 104.488146ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:05:21 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 104.516829ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXXX] duration: 5.000843773s trace: https://jaeger.dev.storj.io/trace/7dda374045a5c6c0 FAIL (2026-03-18 21:05:04 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000949962s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 115.604612ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:04:54 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 115.639464ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXXX] duration: 5.002043854s trace: https://jaeger.dev.storj.io/trace/704a5197fb94c752 FAIL (2026-03-18 21:04:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002137189s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 86.975107ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:03:50 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 87.000432ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXXX] duration: 5.004907568s trace: https://jaeger.dev.storj.io/trace/7a6c9a01eb1a91b7 FAIL (2026-03-18 21:03:48 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.004943448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.598948588s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 21:03:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.598981772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XXX] duration: 5.001150184s trace: https://jaeger.dev.storj.io/trace/59eef68ba3b1381d FAIL (2026-03-18 21:03:23 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00117528s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 95.584753ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:03:13 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 95.683973ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 8.884940279s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 21:02:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.88497369s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 85.186631ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:02:25 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 85.21427ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.202023232s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 21:02:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.202041914s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................XX] duration: 5.000878623s trace: https://jaeger.dev.storj.io/trace/3a4ec265020e6a09 FAIL (2026-03-18 21:02:18 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000901858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................X] duration: 5.002578598s trace: https://jaeger.dev.storj.io/trace/65968918b17eb2e7 FAIL (2026-03-18 21:01:56 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00267672s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 87.266812ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:01:51 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 87.325412ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 4.655432625s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 21:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.655466739s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 96.37253ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:01:22 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 96.400556ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X................] duration: 4.969708803s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 21:01:12 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.969734127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 5.442809967s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 21:01:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.44290729s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 107.967842ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:00:55 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 108.073782ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...............] duration: 3.584215233s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 21:00:46 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584243958s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 4.482417963s remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:00:37 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 4.482462299s core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..............] duration: 1.926508049s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 21:00:22 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.926546408s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 10.04734972s remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:09 GMT Content-Length=132 Cf-Ray=9de72a10bb751c6f-EWR Content-Type=text/html Set-Cookie=_cfuvid=KcxnbBbLYgYkqrI8WanheuzZx1c3ZulvqDd6A15lHlI-1773867599.4719718-1.0.1.1-Olqpr1dr1QoXy5WTC10uUDkcdZCF37zd_7DYNw_cUCc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Cf-Cache-Status=BYPASS Server=cloudflare Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/71aeb6bf5e66dbaf FAIL (2026-03-18 21:00:09 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 10.047374579s core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X..X......................................................] duration: 455.918566ms trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:59:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 455.957764ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.............] duration: 712.471975ms trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:59:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 712.569624ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X............] duration: 1.604940541s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:58:46 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.604959245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.X..X.....................................................] duration: 430.923769ms trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:58:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.959649ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...........] duration: 2.906806566s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:57:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.906838051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.301346487s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:57:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.30137714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXX.X..X....................................................] duration: 3.257770728s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:57:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.257921168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.780488458s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:56:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.780576056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..........] duration: 3.182633495s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:56:36 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.182659335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XXX.X..X...................................................] duration: 2.786353011s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.786452496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.080921332s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:56:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.080950354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.XXX.X..X..................................................] duration: 3.717860292s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:56:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.717896024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.XXX.X..X.................................................] duration: 3.085422767s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:55:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.08546637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.266867277s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:55:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.26695285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.........] duration: 3.838305655s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:55:28 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.838332104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X.XXX.X..X................................................] duration: 3.334668488s trace: https://jaeger.dev.storj.io/trace/68ad04ba5c601def FAIL (2026-03-18 20:55:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:18:48.572742018 (297h18m46.169513049s) [upload=332998]: Starting upload Mar 18 20:18:48.572814133 (297h18m46.169585148s) [upload=332998]: Done starting upload Mar 18 20:18:48.573205245 (297h18m46.169976255s) [upload=332998]: Uploading segments... Mar 18 20:18:48.573282954 (297h18m46.170053987s) [upload=332998]: Got next segment. Inline: false Mar 18 20:18:48.573333166 (297h18m46.170104193s) [upload=332998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:18:48.573340056 (297h18m46.170111079s) [upload=332998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:18:48.598230071 (297h18m46.195001105s) [seg_pos={0 0}, upload=332998]: Begin upload segment... Mar 18 20:18:48.598329962 (297h18m46.195100995s) [seg_pos={0 0}, upload=332998]: Done begin upload segment. Mar 18 20:18:48.598402463 (297h18m46.195173497s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.598427729 (297h18m46.195198759s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599105047 (297h18m46.19587608s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599228530 (297h18m46.195999564s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599283003 (297h18m46.196054038s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599336402 (297h18m46.196107425s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599501357 (297h18m46.196272398s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599582581 (297h18m46.196353611s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.599634938 (297h18m46.196405961s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603539977 (297h18m46.200311012s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.603560925 (297h18m46.200331949s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.604444826 (297h18m46.20121584s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605329522 (297h18m46.202100546s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.605570961 (297h18m46.202341978s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.606223568 (297h18m46.202994612s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.607056417 (297h18m46.203827447s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Uploading piece... Mar 18 20:18:48.674384391 (297h18m46.271155418s) [upload=332998]: Next returned nil segment Mar 18 20:18:48.675167816 (297h18m46.271938851s) [upload=332998]: Waiting for error group managing segments... Mar 18 20:18:48.681652137 (297h18m46.27842316s) [noise=true , address=79.127.219.46:4890 , piece_id=LXHBFKBEIQ6LH2H7EDFOLYXUQHKTMW67UELALJKOWDAJK3FK2NTA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.681693900 (297h18m46.278464927s) [noise=true , address=45.91.116.4:4952 , piece_id=KVRMFYNQPEDLT5WCOMBQMYAZUVVGODWD3EHMF5WMTWFFE3IKPAJQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685213847 (297h18m46.281984874s) [noise=true , address=45.91.116.11:4117 , piece_id=GSQKXK7EMSV4YOV7NTRZRL6Q7ZWKIUSTJ3KMI7HEDQI56QIMUGYQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.685635466 (297h18m46.282406505s) [noise=true , address=77.90.31.16:4736 , piece_id=4GTZGZDBCGLEZIWK5DGABUULONOBAIVRNGMWEM5AMWQ4I6P3QOHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686201703 (297h18m46.282972726s) [noise=true , address=77.90.31.15:4701 , piece_id=JHYPRBOPJP4C4YFKCW5TIUGUHQJIQGCI2THI2ZDDC6OT7332DAZA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.686661563 (297h18m46.283432587s) [noise=true , address=79.127.219.45:4853 , piece_id=C44VFT24VBUQR5XB2JMLHWMIZQIR6ROQXLYX625ZS4DJCILYWNCA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.687161933 (297h18m46.28393296s) [noise=true , address=45.91.116.9:4264 , piece_id=6AF6UOLHT2MA4L5BC4XKE3B4PPDITPW45YEBNCLWZGYZTITJ62GA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688041155 (297h18m46.284812179s) [noise=true , address=79.127.219.38:4822 , piece_id=OFWSNWBFVZSRMIGZT2XUMQN4NJQUPFAMMOHFMSPBSBV2I5YGE5TA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688514321 (297h18m46.285285346s) [noise=true , address=77.90.31.12:4594 , piece_id=A2BJ3RXLDNF7CV7PDNZ5E4CPWW7A5WJDPQEGH5OZHA5UEP5BTVOQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.688944200 (297h18m46.285715224s) [noise=true , address=77.90.31.18:4808 , piece_id=JMNDZLBZNYWSMFCWDWTVXJ2KMHBPOXUERJGPKQ5MUQCYVQHX6KHQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.697835117 (297h18m46.294606141s) [noise=true , address=77.90.31.19:4841 , piece_id=UKLLTOMDAATH7BLPRO2PPBBVXH5Z6EYKLPKQVUTJP2CKLD7V73WA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:18:48.708841802 (297h18m46.305612826s) [noise=true , address=77.90.31.13:4629 , piece_id=TPSPZ2OXW6GNYVO2URYZJLCRSW4BS55GCZQTIQG3GZ72YDZZOOBQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: <nil> Mar 18 20:19:18.539892805 (297h19m16.136663837s) [noise=true , address=145.79.227.41:4085 , piece_id=EBILNDA33EU3DM7Q7BZA2JSQYIMT3UIQC6UZNKQ2CPSTYWK3LWMQ, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:19:18.540084747 (297h19m16.136855779s) [noise=true , address=145.79.227.42:4126 , piece_id=Y7XOMLSD7DRZWM55QEOZ2JZ2K7CWYTGZ3UKKDXITFAANKZKBRF7A, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540135591 (297h19m16.136906612s) [noise=true , address=145.79.227.24:4586 , piece_id=P7PWKVEVT25IAKLQAZREDIZBFDZTAGCBGPCSNHSQU3MKYYGYPECA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540455017 (297h19m16.13722605s) [noise=true , address=145.79.227.29:4945 , piece_id=ET4AIDCE2E44N5UREXC4TR5LY7NPPPMAAEWF4PLZ54DIM474SUKA, seg_pos={0 0}, upload=332998]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:19:18.540487128 (297h19m16.137258142s) [seg_pos={0 0}, upload=332998]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:19:18.560042117 (297h19m16.15681314s) [upload=332998]: Done uploading segments... Mar 18 20:19:18.560096823 (297h19m16.156867851s) [upload=332998]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.334735686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXX................XXXXXXXXXXXXXXXXXXXXXXXXXXXX.X........] duration: 3.31467748s trace: https://jaeger.dev.storj.io/trace/1929f8251959ebe2 FAIL (2026-03-18 20:54:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.314703485s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)